action-build-and-upload-conda-packages

⚙️ Build your conda package
🔄 Convert your conda package
🚀 Upload your conda package
✅ Completely automated!!

Content

• About
• Requirements
  • Conda-build recipe
  • Conda base environment
  • Anaconda token
    • Create an Anaconda token
      • Using the command line
      • Through the Anaconda.org website
    • Add the Anaconda token to the GitHub Secrets
• How to use
• Input parameters
  • Additional command line arguments
• Outputs
• Examples
• Acknowledgements

About

This GitHub Action automates the process of building, converting, and uploading Conda packages to Anaconda.org. It streamlines package distribution by handling:

• Building: Uses conda build to create Conda packages from a conda-build recipe.
• Converting: Utilizes conda convert to generate platform-specific package variants.
• Uploading: Publishes the final package versions to Anaconda.org for easy distribution.

By integrating this action into your CI/CD GitHub workflow, you can ensure that your Conda packages are consistently built, transformed for multiple platforms, and made available to your users with minimal manual effort.

This GitHub Action was originally developed by the Computational Biology and Drug Design Research Unit (UIBCDF) at the Mexico City Children's Hospital Federico Gómez. For the complete list of contributors, refer to the contributors section.
Explore more GitHub Actions developed by UIBCDF at the UIBCDF GitHub Organization page.

Requirements

Conda-build recipe

A conda-build recipe defines the instructions for building your package, primarily through a meta.yaml file, optionally accompanied by other supporting files.
These files can be placed within a .conda directory in your repository.
For details on how to structure a conda-build recipe, refer to the Conda metadata instructions.

Conda base environment

This action relies on conda commands, which must be accessible within the GitHub runner. To ensure this, a base Conda environment needs to be set up.
We recommend using the conda-incubator/setup-miniconda action in a preceding step of your GitHub workflow to configure the Conda base environment:

steps:
      ...
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 
          environment-file: path/to/conda/env.yaml # Path to the conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      ...      
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        ...

Anaconda token

In order to upload a package to your anaconda user or organization channel, you need to create an Anaconda token.

Create an Anaconda token

There are two main ways to create an Anaconda token:

• Using the command line
• Through the Anaconda.org website

Using the command line

You can create an Anaconda token from your terminal, using anaconda auth:

# Replace 'MyToken' with the name you want for your token
anaconda auth --create --name MyToken

Tip

To create token for an Anaconda Organization, add the --org option to the command above:

# Replace 'MyOrg' with the name of the organization
# Replace 'MyToken' with the name you want for your token
anaconda auth --create --name MyToken --org MyOrg

Through the Anaconda.org website

1. Log in to Anaconda.org
2. From your profile in the top-right corner, select Settings.
3. Click Access in the left-hand menu.
4. Fill out the Create access token form:
   1. Provide a unique token name.
   2. Set your token strength to strong (longer token). This generates a strong, completely unique token that is difficult to guess with brute force methods.
   3. Set the required scopes for your use case.
   4. Set the expiration date.
5. Click on Create

Tip

To create a token for an Anaconda Organization, follow the steps outlined in Issuing/reissuing a token.

Add the Anaconda token to the GitHub Secrets

The best practice for using an Anaconda token in a GitHub Action is to store it as a GitHub Secret. This helps keep the token secure and prevents it from being exposed in workflow logs.
For instructions on creating and using GitHub Secrets, refer to GitHub's documentation.

How to use

You can include this GitHub Action as a step within a GitHub workflow, placed in the .github/workflows directory within your repository.

An example of the basic usage of this GitHub Action is displayed below:

name: Build and upload conda packages

on:
  push:
    branches: main

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret

Input parameters

Name	Description	Required/Optional	Default value
meta_yaml_dir	Path to the directory where the meta.yaml file is located.	Required
upload	Upload the built package to Anaconda. If set to false, the built package will not be uploaded to Anaconda.org.	Optional	true
overwrite	Do not abort the uploading if a package with the same name is already present in the Anaconda channel.	Optional	false
mambabuild	Uses conda mambabuild command to build the packages. Requires mamba setup.	Optional	false
user	Name of the Anaconda.org channel where the package will be uploaded.	Optional
token	Anaconda token for the package uploading.	Optional
label	Label of the uploaded package.	Optional	main
platform_host	Build packages for the host platform.	Optional	true
platform_all	Build packages for all supported platforms.	Optional	false
platform_linux-64	Build packages for the linux-64 platform.	Optional	false
platform_linux-32	Build packages for the linux-32 platform.	Optional	false
platform_osx-64	Build packages for the osx-64 platform.	Optional	false
platform_osx-arm64	Build packages for the osx-arm64 platform.	Optional	false
platform_linux-ppc64	Build packages for the linux-ppc64 platform.	Optional	false
platform_linux-ppc64le	Build packages for the linux-ppc64le platform.	Optional	false
platform_linux-s390x	Build packages for the linux-s390x platform.	Optional	false
platform_linux-armv6l	Build packages for the linux-armv6l platform.	Optional	false
platform_linux-armv7l	Build packages for the linux-armv7l platform.	Optional	false
platform_linux-aarch64	Build packages for the linux-aarch64 platform.	Optional	false
platform_win-32	Build packages for the linux-win-32 platform.	Optional	false
platform_win-64	Build packages for the linux-win-64 platform.	Optional	false
conda_build_args	Additional command line arguments to pass to the conda build command.	Optional
conda_convert_args	Additional command line arguments to pass to the conda convert command.	Optional
anaconda_upload_args	Additional command line arguments to pass to the anaconda upload command.	Optional

Additional command line arguments

This action, internally, calls the following commands:

• conda build (or conda mambabuild, if mambabuild is set to true)
• conda convert (if any platform conversion is specified)
• anaconda upload (if upload is set to true)

The above commands have multiple command-line arguments that can be passed, for each of the respective command, by using the conda_build_args, conda_convert_args and anaconda_upload_args input parameters.

Refer to the Pass additional command-line arguments example for a practical case on the usage of these input parameters.

Warning

Some command line arguments in conda_build_args, conda_convert_args or anaconda_upload_args cannot be specified because they are either already handled internally by the action or conflict with specific input parameters:
conda_build_args

• --output-folder
• --no-anaconda-upload

conda_convert_args

• --output-folder

anaconda_upload_args

• --label/-l together with the label input parameter
• --user/-u together with the user input parameter
• --force together with the overwrite input parameter

Outputs

Name	Description
paths	Space-separated paths for the built packages, in the format path1 path2 ... pathN.

The output paths can be useful for later jobs, for example to create a GitHub release with the built packages as artifacs.

Examples

Build and upload a package when a new Git tag is pushed and use the Git tag as the package version

The meta.yaml file defines the package version field to specify the version number of the built package.

To automatically set the package version to the latest Git tag, Jinja Templating can be used by setting the version value to {{ GIT_DESCRIBE_TAG }} in the meta.yaml file:

# meta.yaml file
package:
  name: your_package_name # replace with your package name
  version: {{ GIT_DESCRIBE_TAG }}

Then, to build and upload a package whenever a new Git tag is pushed, your workflow yaml file can look like the following:

# your_worfklow.yaml file
name: Build and upload conda packages when a tag gets pushed

on:
  push:
    tags: 
      - '*'

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
        with:
          fetch-tags: true
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret

💡TIP
Version numbers including the dash character - are not supported by conda-build. If you want to release a version like 1.0.0-beta.1, replace it with something like 1.0.0b1.

Build a pure Python conda package for different platforms

When a package is built as pure Python library, conda convert can generate packages for other platforms.

To create packages for multiple platforms (such as 'linux-64', 'osx-64' and 'win-64`), you can use the following workflow:

# your_worfklow.yaml file
name: Build and upload pure-python conda package for different platforms

on:
  push:
    branches:
      - main

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret
          platform_linux-64: true
          platform_osx-64: true
          platform_win-64: true

Build a platform-specific conda package for different platforms

If a package requires platform-specific compilation instructions, conda convert is not a viable option.

Instead, the package can be built separately for each target platform by running multiple Conda builds in parallel using the GitHub matrix strategy:

# your_worfklow.yaml file
name: Build and upload platform-specific conda packages

on:
  push:
    branches:
      - main

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [macos-latest, ubuntu-latest, windows-latest]
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret

This setup will run three jobs in parallel, building and uploading the Conda package for the macos-latest, ubuntu-latest and windows-latest platforms.

💡TIP
In this case, your meta.yaml file will likely need preprocessing selectors to differenciate builds across platforms.

Pass additional command-line arguments

To build a package and limit the search for dependencies to specific Anaconda channels, the --override-channels and --channel my_channel options can be passed to the to the conda build command.

Additionally, to display Python imports for the compiled components of the built package, the --show-imports option can be passed to the conda convert command.

To apply these options when building and uploading a package, you can use the following workflow:

# your_worfklow.yaml file
name: Build and upload conda packages with specific Anaconda channels and showing Python imports

on:
  push:
    branches:
      - main

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret
          platform_osx-64: true
          conda_build_args: --override-chanels --channel my_channel # Replace my_channel with the name of the specific channel
          conda_convert_args: --show-imports

Build a package for multiple Python versions

The recommended approach for building packages across different Python versions is to use build variants.

This involves placing a conda_build_config.yaml file inside the conda recipe directory, specifying variant inputs. conda build will then generate a separate build for each variant.

For example, to build a package for Python 2.7 and 3.5, we can define:

# conda_build_config.yaml file
python:
    - 2.7
    - 3.5

# meta.yaml file
...
package:
    name: your_package_name # Replace with your package name
    version: package_version # Replace with your package version

requirements:
    build:
        - python
    run:
        - python
...

In this case, no changes are needed in the workflow file for this action. A setup similar to the basic example will work.

Set the package label depending on the release type

name: Build and upload conda packages with label according to release type

on:
  release:
    types: [released, prereleased]

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Set label
        id: set-label
        shell: bash
        run: |
          if [[ "${{ github.event.action }}" == "prereleased" ]]; then
            label=dev
          else
            label=main
          echo "label=$label" >> $GITHUB_OUTPUT
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret
          label: ${{ steps.set-label.outputs.label }}

Create a GitHub release with the built packages as artifacs

name: Build and upload conda packages and create GitHub release with built packages as artifacts

on:
  push:
    tags: main

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        id: conda-build-and-upload
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret
      - name: Re-format output paths
        id: reformat-paths
        # Needed to have the correct newline-separated files format for the following release step
        run: |
            paths=$(tr ' ' '\n' <<< "${{steps.conda-build-and-upload.outputs.paths}}")
            echo "newline-separated-paths=$paths" >> $GITHUB_OUTPUT
      - name: Create GitHub release
        uses: softprops/action-gh-release@v2
        with:
            tag_name: ${{ github.ref_name }}
            name: your_release_name # Replace with the name for your release
            generate_release_notes: true
            fail_on_unmatched_files: true
            files: ${{steps.reformat-paths.outputs.newline-separated-paths}}

Test correctness of a Conda build

You can use this action in a CI/CD workflow to test for errors during the conda build and conversion process, without uploading the package to Anaconda.org.

To do this, set upload: false as an input parameter for the action:

name: Test conda build

on:
  pull_request:
    branches: main

jobs:
  conda_deployment:
    name: Conda deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: 3.11
          environment-file: path/to/conda/env.yaml    # Replace with the path to your conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta.yaml/directory # Replace with the path to your meta.yaml directory
          user: uibcdf # Replace with your Anaconda username (or an Anaconda organization username)
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the name of your Anaconda Token secret
          upload: false

Acknowledgements

This GitHub Action was initially developed to address a specific need of the UIBCDF and serve as an example of an in-house GitHub Action for its researchers and students.

We extend our gratitude to the developers of the following related GitHub Actions, whose work provided valuable insights and guidance in setting up our own.

https://github.com/fdiblen/anaconda-action https://github.com/MichaelsJP/conda-package-publish-action https://github.com/Jegp/conda-package-publish-action https://github.com/amauryval/publish_conda_package_action https://github.com/elbeejay/conda-publish-action https://github.com/maxibor/conda-package-publish-action https://github.com/m0nhawk/conda-package-publish-action https://github.com/fcakyon/conda-publish-action