Add GH Action to build and publish Docker images

[trsvchn]

Fixes #1305

Description:

Adds GitHub Action that triggers on push to master docker folder or releases to build and publish Docker images

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[vfdev-5]

@trsvchn thanks a lot for the PR !
I wonder if you could test it somehow on your fork or somewhere else ?
When specified runs-on: ubuntu-latest does it have docker installed to build images ?
A reminder for myself: create secrets for username/tokens

I propose to add temporary on: pull-request, after buliding images, add a command like docker images to list all built images. Finally, try to publish with correct username and incorrect token (set by me) such that it should only fail on pushing. Such that we can "ensure" correctness of the GH action. What do you think ?

[trsvchn]

@vfdev-5 Sure, I am going to try it on my fork

When specified runs-on: ubuntu-latest does it have docker installed to build images ?

Yes, It has docker preinstalled, since jobs run inside a virtual machine, here is I created demo repo,:

https://github.com/trsvchn/docker-playground-action/runs/1274842834?check_suite_focus=true#step:3:1

In addition, there are some docker images already:

https://github.com/trsvchn/docker-playground-action/runs/1274842834?check_suite_focus=true#step:4:1

You just need to use runs-on: ubuntu-latest.

You can run docker container inside VM, like this:

  hello-docker-container:
    name: Try to run container
    runs-on: ubuntu-latest
    container: python:3

Here is the workflow file

[vfdev-5]

@trsvchn sounds good ! Thanks the info, good to know that we are nothing to setup to run with a docker 👍

[trsvchn]

I propose to add temporary on: pull-request, after buliding images, add a command like docker images to list all built images. Finally, try to publish with correct username and incorrect token (set by me) such that it should only fail on pushing. Such that we can "ensure" correctness of the GH action. What do you think ?

I am experimenting inside my fork, and this is nice, because I can turn-off all other workflows and work only with docker workflow.

[trsvchn]

UPDATE: run: cd docker doesn't work, you need use working-directory: ./docker on each step.

[trsvchn]

OK, the action is configured but Horovod build failed:
https://github.com/trsvchn/ignite/runs/1275008864?check_suite_focus=true#step:3:1615

[vfdev-5]

OK, the action is configured

Nice !

@trsvchn let me check what is the issue there with Horovod build.

[trsvchn]

Main fails as well:

https://github.com/trsvchn/ignite/runs/1275075846?check_suite_focus=true#step:3:439

[vfdev-5]

Yeah, just saw it :)

++ docker run --rm -it pytorchignite/base:latest -c 'python -c "import torch; import ignite; print(torch.version + "-" + ignite.version, end="")"'
the input device is not a TTY

Probably, we can find another more proper way to fetch the version :)

[trsvchn]

Probably, we can find another more proper way to fetch the version :)

Yeah, definitely) same for msdp

++ docker run --rm -it pytorchignite/msdp-apex-base:latest -c 'python -c "import torch; import ignite; print(torch.version + "-" + ignite.version, end="")"'
the input device is not a TTY

• image_tag=
  Error: Process completed with exit code 1.

[trsvchn]

@vfdev-5 I think we just need to use -i instead of -it

[vfdev-5]

@trsvchn I left a comment to the latest commit about removing one of -c. Maybe, to iterate faster you can disable building part and check docker run --rm -i pytorchignite/${image_name}:latest ..., as there is a version on hub.docker it could pull the latest version.

Seems like main is built and the fail is at push

[trsvchn]

@vfdev-5 -i did the thing!

https://github.com/trsvchn/ignite/runs/1275326281?check_suite_focus=true#step:4:1

docker images | grep pytorchignite
  shell: /bin/bash -e {0}
pytorchignite/apex-nlp                               1.6.0-0.4.2                   cb179b3a5c46        Less than a second ago   3.87GB
pytorchignite/apex-nlp                               latest                        cb179b3a5c46        Less than a second ago   3.87GB
pytorchignite/apex-vision                            1.6.0-0.4.2                   798acfbca85e        21 seconds ago           4.38GB
pytorchignite/apex-vision                            latest                        798acfbca85e        21 seconds ago           4.38GB
pytorchignite/apex                                   1.6.0-0.4.2                   2e30b5a737e6        About a minute ago       3.73GB
pytorchignite/apex                                   latest                        2e30b5a737e6        About a minute ago       3.73GB
pytorchignite/nlp                                    1.6.0-0.4.2                   77cc4d16ea86        14 minutes ago           3.82GB
pytorchignite/nlp                                    latest                        77cc4d16ea86        14 minutes ago           3.82GB
pytorchignite/vision                                 1.6.0-0.4.2                   f7a6f1db205a        14 minutes ago           4.33GB
pytorchignite/vision                                 latest                        f7a6f1db205a        14 minutes ago           4.33GB
pytorchignite/base                                   1.6.0-0.4.2                   97f5a9931a28        16 minutes ago           3.68GB
pytorchignite/base                                   latest                        97f5a9931a28        16 minutes ago           3.68GB

[trsvchn]

I read your comment, this works fine:

image_tag=`docker run --rm -i pytorchignite/${image_name}:latest -c 'python -c "import torch; import ignite; print(torch.__version__ + \"-\" + ignite.__version__, end=\"\")"'`

[vfdev-5]

I read your comment, this works fine:

image_tag=`docker run --rm -i pytorchignite/${image_name}:latest -c 'python -c "import torch; import ignite; print(torch.__version__ + \"-\" + ignite.__version__, end=\"\")"'`

Yes, let's keep it like that. I checked also locally and without the first -c it does not work. So, let's keep it as you did.

[trsvchn]

I read your comment, this works fine:

image_tag=`docker run --rm -i pytorchignite/${image_name}:latest -c 'python -c "import torch; import ignite; print(torch.__version__ + \"-\" + ignite.__version__, end=\"\")"'`

Yes, let's keep it like that. I checked also locally and without the first -c it does not work. So, let's keep it as you did.

yes here is a proof)

Or you can change it to -i if you have input piped into the docker command that doesn't come from a TTY. If you have something like xyz | docker ... or docker ... <input in your command line, do this.

Error “The input device is not a TTY”

[trsvchn]

@vfdev-5 So, I am going to update the PR with -i and configured workflow, but I will comment out the horovod for now and I will add on: pull_request to check the action in action)

[vfdev-5]

@trsvchn could you try to build locally a single horovod image on CPU only infra, please ? If you have a similar issue than probably we can not build horovod without a GPU...

@vfdev-5 So, I am going to update the PR with -i and configured workflow, but I will comment out the horovod for now and I will add on: pull_request to check the action in action)

Sounds good !

[trsvchn]

@trsvchn could you try to build locally a single horovod image on CPU only infra, please ? If you have a similar issue than probably we can not build horovod without a GPU...

Yes! I have a such machine, but it takes some time)

[vfdev-5]

I also added secrets to use with push

steps:
  - name: Hello world action
    env: # Or as an environment variable
      DOCKER_USER: ${{ secrets.DOCKER_USER }}
      DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }}

Currently, the token is set incorrect to check the bash script.

[trsvchn]

I also added secrets to use with push

Great! Can I add them to yml?

[vfdev-5]

I also added secrets to use with push

Great! Can I add them to yml?

Yes, please, it probably should be like the code above.

[trsvchn]

@vfdev-5 I've got the same issue for Horovod on the local CPU-only machine:

On GH Action VM:

Building wheels for collected packages: horovod, psutil, pyyaml
  Building wheel for horovod (setup.py): started
  Building wheel for horovod (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-oxm96qf0/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-oxm96qf0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-wzpzp2o8
       cwd: /tmp/pip-req-build-oxm96qf0/
Complete output (1071 lines):

...

  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
  Building wheel for psutil (setup.py): started
  Building wheel for psutil (setup.py): finished with status 'done'
  Created wheel for psutil: filename=psutil-5.7.2-cp37-cp37m-linux_x86_64.whl size=283600 sha256=f25940592581464e8aa4df7b656ab65e0f705724a328898ead7b8d98d692e34d
  Stored in directory: /tmp/pip-ephem-wheel-cache-td4yfx19/wheels/2d/43/97/00701864a7bee6d9e1a52dd682537dcbf1d013d0e2e6f0c1f1
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44619 sha256=7a77d79a9bf41dfeb91eadb3523a249b613f07762cad6c2c02b364fa5fac70e6
  Stored in directory: /tmp/pip-ephem-wheel-cache-td4yfx19/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
Successfully built psutil pyyaml
Failed to build horovod
ERROR: Failed to build one or more wheels
The command '/bin/sh -c apt-get update && apt-get install -y git &&         git clone --recursive --depth 1 --branch v0.20.0 https://github.com/horovod/horovod.git /horovod &&         conda install -y cmake=3.16 nccl=2.7 -c conda-forge &&         cd /horovod &&         HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_NCCL_LINK=SHARED HOROVOD_WITHOUT_MPI=1 HOROVOD_WITH_PYTORCH=1 pip wheel --no-cache-dir . &&         rm -rf /var/lib/apt/lists/*' returned a non-zero code: 1

On Local CPU-only machine:

Building wheels for collected packages: horovod, psutil, pyyaml
  Building wheel for horovod (setup.py): started
  Building wheel for horovod (setup.py): still running...
  Building wheel for horovod (setup.py): still running...
  Building wheel for horovod (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-l59s03l0/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-l59s03l0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-jnlf3nk7
       cwd: /tmp/pip-req-build-l59s03l0/
  Complete output (1071 lines):

...

  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
  Building wheel for psutil (setup.py): started
  Building wheel for psutil (setup.py): finished with status 'done'
  Created wheel for psutil: filename=psutil-5.7.2-cp37-cp37m-linux_x86_64.whl size=283599 sha256=abbe06e81ff93895397fb1b9f69e9b1b50852ef72ea21c2752fec635d093b8b3
  Stored in directory: /tmp/pip-ephem-wheel-cache-6ltpp9lt/wheels/2d/43/97/00701864a7bee6d9e1a52dd682537dcbf1d013d0e2e6f0c1f1
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44619 sha256=8f11f852a6ac2f25a88dc82baf24ccf0bd94cad7d8bd926b73bd68a4b2fba7b7
  Stored in directory: /tmp/pip-ephem-wheel-cache-6ltpp9lt/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
Successfully built psutil pyyaml
Failed to build horovod
ERROR: Failed to build one or more wheels
The command '/bin/sh -c apt-get update && apt-get install -y git &&         git clone --recursive --depth 1 --branch v0.20.0 https://github.com/horovod/horovod.git /horovod &&         conda install -y cmake=3.16 nccl=2.7 -c conda-forge &&         cd /horovod &&         HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_NCCL_LINK=SHARED HOROVOD_WITHOUT_MPI=1 HOROVOD_WITH_PYTORCH=1 pip wheel --no-cache-dir . &&         rm -rf /var/lib/apt/lists/*' returned a non-zero code: 1

[vfdev-5]

@trsvchn thanks a lot for reproducing the issue ! I saw in the logs that the issue is related to generating some cuda kernels: https://github.com/trsvchn/ignite/runs/1275008864?check_suite_focus=true#step:3:961
Let's try to figure out if we could fix it. Otherwise, we have to build it on Circle CI with GPU support... :(

[trsvchn]

@vfdev-5 At least, for now, main and msdp work well, here is logs from latest test on the forked repo

https://github.com/trsvchn/ignite/runs/1276022256?check_suite_focus=true#step:5:1

here push all log (with invalid tokens)

https://github.com/trsvchn/ignite/runs/1276304424?check_suite_focus=true#step:5:9

[vfdev-5]

@trsvchn I tested building hvd-base image on two machines: a) ~cpu only and b) with CUDA_VISIBLE_DEVICES="" docker build .... Both are passing. Let's check again here with updated horovod branch to 0.20.3.

[trsvchn]

@trsvchn I tested building hvd-base image on two machines: a) ~cpu only and b) with CUDA_VISIBLE_DEVICES="" docker build .... Both are passing. Let's check again here with updated horovod branch to 0.20.3.

@vfdev-5
Sounds great! I can test it on my fork, what changes are required?

[vfdev-5]

@trsvchn the diff is

-        git clone --recursive --depth 1 --branch v0.20.0 https://github.com/horovod/horovod.git /horovod && \
+        git clone --recursive --depth 1 --branch v0.20.3 https://github.com/horovod/horovod.git /horovod && \

in

        modified:   docker/hvd/Dockerfile.hvd-apex
        modified:   docker/hvd/Dockerfile.hvd-base

Yeah, let's see if it passes on the CI.

[trsvchn]

@vfdev-5 YES! Now it works!

https://github.com/trsvchn/ignite/runs/1286774229?check_suite_focus=true#step:6:1

[vfdev-5]

That's awesome ! So, let's update this PR. I'm thinking if it is good if we could keep images building on PR but no push to the docker hub and we rebuild them and push on push/master github event. Rebuilding images on push/master github event is not very eco solution though.

[trsvchn]

That's awesome ! So, let's update this PR. I'm thinking if it is good if we could keep images building on PR but no push to the docker hub and we rebuild them and push on push/master github event. Rebuilding images on push/master github event is not very eco solution though.

Yes that makes sense, we can do smth like:

      - if github.event_name == 'push'
        name: Push all PyTorch-Ignite Docker images
        env:
          DOCKER_USER: ${{ secrets.DOCKER_USER }}
          DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }}
        working-directory: ./docker
        run: |
          chmod +x ./push_all.sh
          ./push_all.sh

[vfdev-5]

@trsvchn thanks! Let's do it like that

[trsvchn]

@vfdev-5 I have an idea, what if split this workflow into 3 additional with builds triggered only with changes in specific dirs, for example:

This one just builds Horovod images on PR, push/master and changes in hvd dir:

name: Build only Horovod flavoured PyTorch-Ignite images 

on:
  push:
    branches:
      - master
    paths:
      - docker/hvd/**
  pull_request:
    paths:
      - docker/hvd/**

jobs:
  build:
    name: Build all Horovod flavoured PyTorch-Ignite images
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Build all Horovod flavoured PyTorch-Ignite images
        working-directory: ./docker
        run: |
          chmod +x ./hvd/build_all.sh
          ./hvd/build_all.sh

      - if github.event_name == 'push'
        # push hvd only images

The same for main and msdp. (just builds)

Then we just need to adjust existing workflow to start only on releases, and this one will push to docker hub

name: Build And Publish Docker Images

on:
  release:
    types: [published]

What do you think?

[vfdev-5]

@trsvchn this would be nice, however I do not know how to pass built images from different workflows as we have to upload somewhere the artifacts ?

[trsvchn]

@trsvchn this would be nice, however I do not know how to pass built images from different workflows as we have to upload somewhere the artifacts ?

Yeah, and we have quite large artifacts :(

Yes, proposed solution just decreases number of builds per workflow

[vfdev-5]

Maybe we can reconfigure push_all such that it could push only what we need ?

Btw, CI still does not build images. I was also thinking to reduce the number of running CI actions here : #1269 but seems like the PR on it got stuck

[trsvchn]

Btw, CI still does not build images.

Interesting, and why it works in the forked repo

[vfdev-5]

I think the issue is here

- if github.event_name == 'push'
   name: Push all PyTorch-Ignite Docker images

which probably would look like (to verify)

- name: Push all PyTorch-Ignite Docker images
   if: github.event_name == 'push'

[trsvchn]

I think the issue is here

- if github.event_name == 'push'
   name: Push all PyTorch-Ignite Docker images

which probably would look like (to verify)

- name: Push all PyTorch-Ignite Docker images
   if: github.event_name == 'push'

Yes, I think you are right

[vfdev-5]

@trsvchn Thanks a lot for the PR ! I think it is perfect for the first iteration. I'll merge it once docker token is updated and we can push updated images then. If you'd like to setup 3 workflows to build only modified docker image in another PR, it would be awesome !

[trsvchn]

@trsvchn Thanks a lot for the PR ! I think it is perfect for the first iteration. I'll merge it once docker token is updated and we can push updated images then.

@vfdev-5 Yeah I think it's a good start! :)

If you'd like to setup 3 workflows to build only modified docker image in another PR, it would be awesome !

Sure! I would love to do that!