Add triggering pipelines mini-tutorial #3495

strickvl · 2025-04-04T09:28:31Z

Added a tutorial showing how to trigger pipelines

Everything in there has been tested by me + it works / runs.

- Corrected the formatting of the instructions for setting up GitHub Secrets, ensuring clarity in the steps provided. - Improved the readability of the follow-up actions section by breaking a long sentence into two and ensuring proper line endings. This update aims to enhance user understanding and navigation within the tutorial, providing clearer guidance on integrating external systems with ZenML pipelines.

…lines - Introduced a comprehensive tutorial detailing the setup of CI/CD workflows for ML pipelines using ZenML and GitHub Actions. This includes steps for creating development and production stacks, setting up authentication, and configuring GitHub Actions for both CI and CD. - Updated the Table of Contents to include a new entry for "Continuous Integration and Deployment," enhancing navigation for users seeking guidance on CI/CD practices within ZenML. This addition aims to provide users with a structured approach to automating testing and deployment, ensuring high-quality models are delivered to production while maintaining reproducibility across environments.

…elines-and-ci-cd-tutorials

This commit updates the headers in the trigger-pipelines-from-external-systems.md tutorial to follow a more descriptive, action-oriented style consistent with other tutorials. Headers now focus on what each section accomplishes rather than using generic step numbers. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

This commit simplifies the pipeline example in the trigger-pipelines-from-external-systems.md tutorial by: 1. Removing the redundant save_results step since ZenML already handles artifact serialization 2. Updating the train_model step to return the model directly instead of a dictionary 3. Updating the evaluate step to accept the model directly 4. Removing the output_path parameter from the pipeline function 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

This commit updates the trigger-pipelines-from-external-systems.md tutorial with corrections for the Run Template section, based on verified working code: 1. Improve template creation code to use proper Pipeline ID and run sorting 2. Add detailed example for programmatic template triggering using Python client 3. Clarify that step parameters must be set at the step level in Python client 4. Add note about REST API parameter mapping to step parameters 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

This commit updates the REST API example in the trigger-pipelines-from-external-systems.md tutorial to use the correct payload format for triggering templates: 1. Changed from using "pipeline_parameters" to using step-level parameters 2. Updated the note to clarify that parameters must be specified at the step level 3. Aligned the example with the existing use-templates-rest-api.md documentation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…elines-and-ci-cd-tutorials

- Add run template endpoint to Method 2 custom API example - Update documentation to show both direct pipeline and template execution - Add advantages of using templates in production - Keep consistent parameter handling between methods

Copilot

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

coderabbitai · 2025-04-04T09:28:38Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai plan to trigger planning for file edits and PR creation.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

strickvl · 2025-04-04T09:29:40Z

It includes the CI/CD docs draft but they aren't in the TOC. I can remove if you'd rather keep it out of the repo.

github-actions · 2025-04-04T09:29:47Z

Documentation Link Check Results

✅ Absolute links check passed
✅ Relative links check passed
_{Last checked: 2025-04-10 10:00:49 UTC}

docs/book/user-guide/tutorials/trigger-pipelines-from-external-systems.md

docs/book/user-guide/tutorials/continuous-integration-and-deployment.md

…er guide. This deletion includes all sections related to setting up CI/CD for ML pipelines using ZenML and GitHub Actions, along with example code and best practices. The content was deemed redundant or outdated.

- Added information on creating run templates from existing pipeline runs. - Included examples for passing configuration files and specifying stacks when using the `create-run-template` command. - Emphasised best practices for tagging run templates and maintaining parameter stability when triggering pipelines externally. - Updated security best practices section to include API key usage. This update aims to improve clarity and usability for users integrating external systems with ZenML pipelines.

htahir1

I left some comments - we need to talk sync about this

docs/book/user-guide/tutorials/trigger-pipelines-from-external-systems.md

….com:zenml-io/zenml into doc/triggering-pipelines-and-ci-cd-tutorials

…elines-and-ci-cd-tutorials

… implementation steps This update provides comprehensive instructions for building a custom pipeline trigger API, including code examples, Dockerfile setup, and deployment guidelines. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Revised heading structure in the tutorial to maintain consistent style between methods. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Copilot

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

strickvl · 2025-04-09T15:30:15Z

@htahir1 updated as per your feedback. Tested that it works when the API is deployed.

strickvl · 2025-04-09T15:31:14Z

Preview the docs here

docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md

The tutorial now clearly explains that PIPELINE_API_KEY is a security token to protect API endpoints and emphasizes the importance of using strong, random keys in production environments. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

….com:zenml-io/zenml into doc/triggering-pipelines-and-ci-cd-tutorials

htahir1

Almost there!

docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md

- Add status_code=202 to the /trigger-async endpoint to properly indicate accepted status - Add status_code=200 to the /trigger endpoint for consistency - Better aligns API behavior with REST standards 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Remove synchronous /trigger endpoint - Use only one asynchronous trigger endpoint for better comparison with Method 1 - Update documentation to reflect the API changes - Streamline the Triggering Pipelines via API section 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

* WIP v1 * Update user guide tutorial on triggering pipelines from external systems - Corrected the formatting of the instructions for setting up GitHub Secrets, ensuring clarity in the steps provided. - Improved the readability of the follow-up actions section by breaking a long sentence into two and ensuring proper line endings. This update aims to enhance user understanding and navigation within the tutorial, providing clearer guidance on integrating external systems with ZenML pipelines. * WIP Add tutorial on Continuous Integration and Deployment for ML Pipelines - Introduced a comprehensive tutorial detailing the setup of CI/CD workflows for ML pipelines using ZenML and GitHub Actions. This includes steps for creating development and production stacks, setting up authentication, and configuring GitHub Actions for both CI and CD. - Updated the Table of Contents to include a new entry for "Continuous Integration and Deployment," enhancing navigation for users seeking guidance on CI/CD practices within ZenML. This addition aims to provide users with a structured approach to automating testing and deployment, ensuring high-quality models are delivered to production while maintaining reproducibility across environments. * Update headers * docs: improve tutorial header styles for consistency This commit updates the headers in the trigger-pipelines-from-external-systems.md tutorial to follow a more descriptive, action-oriented style consistent with other tutorials. Headers now focus on what each section accomplishes rather than using generic step numbers. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * docs: simplify trigger pipelines tutorial This commit simplifies the pipeline example in the trigger-pipelines-from-external-systems.md tutorial by: 1. Removing the redundant save_results step since ZenML already handles artifact serialization 2. Updating the train_model step to return the model directly instead of a dictionary 3. Updating the evaluate step to accept the model directly 4. Removing the output_path parameter from the pipeline function 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * docs: update run template usage in pipeline trigger tutorial This commit updates the trigger-pipelines-from-external-systems.md tutorial with corrections for the Run Template section, based on verified working code: 1. Improve template creation code to use proper Pipeline ID and run sorting 2. Add detailed example for programmatic template triggering using Python client 3. Clarify that step parameters must be set at the step level in Python client 4. Add note about REST API parameter mapping to step parameters 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * docs: fix REST API template payload in pipeline trigger tutorial This commit updates the REST API example in the trigger-pipelines-from-external-systems.md tutorial to use the correct payload format for triggering templates: 1. Changed from using "pipeline_parameters" to using step-level parameters 2. Updated the note to clarify that parameters must be specified at the step level 3. Aligned the example with the existing use-templates-rest-api.md documentation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Updates to triggers section * docs: add run template support to custom API tutorial - Add run template endpoint to Method 2 custom API example - Update documentation to show both direct pipeline and template execution - Add advantages of using templates in production - Keep consistent parameter handling between methods * add links * Fix CLI docs formatting * Remove the continuous integration and deployment tutorial from the user guide. This deletion includes all sections related to setting up CI/CD for ML pipelines using ZenML and GitHub Actions, along with example code and best practices. The content was deemed redundant or outdated. * docs: enhance trigger pipelines tutorial with run template details - Added information on creating run templates from existing pipeline runs. - Included examples for passing configuration files and specifying stacks when using the `create-run-template` command. - Emphasised best practices for tagging run templates and maintaining parameter stability when triggering pipelines externally. - Updated security best practices section to include API key usage. This update aims to improve clarity and usability for users integrating external systems with ZenML pipelines. * Updates following review * Expand Method 2 section in pipeline triggering tutorial with detailed implementation steps This update provides comprehensive instructions for building a custom pipeline trigger API, including code examples, Dockerfile setup, and deployment guidelines. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Update Method 2 section headings to match Method 1 style Revised heading structure in the tutorial to maintain consistent style between methods. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * add icon for page * Add explicit explanation of PIPELINE_API_KEY security purpose The tutorial now clearly explains that PIPELINE_API_KEY is a security token to protect API endpoints and emphasizes the importance of using strong, random keys in production environments. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Add explicit status codes to FastAPI endpoints - Add status_code=202 to the /trigger-async endpoint to properly indicate accepted status - Add status_code=200 to the /trigger endpoint for consistency - Better aligns API behavior with REST standards 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Simplify custom API section in pipeline trigger tutorial - Remove synchronous /trigger endpoint - Use only one asynchronous trigger endpoint for better comparison with Method 1 - Update documentation to reflect the API changes - Streamline the Triggering Pipelines via API section 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]> (cherry picked from commit 7ec8c6f)

strickvl and others added 13 commits April 1, 2025 19:20

WIP v1

5c4e74c

Merge remote-tracking branch 'origin/develop' into doc/triggering-pip…

90addb7

…elines-and-ci-cd-tutorials

Update headers

17ce51b

Updates to triggers section

d038e71

Merge remote-tracking branch 'origin/develop' into doc/triggering-pip…

b3c085f

…elines-and-ci-cd-tutorials

add links

b3922be

strickvl added documentation Improvements or additions to documentation internal To filter out internal PRs and issues labels Apr 4, 2025

strickvl requested review from htahir1 and Copilot April 4, 2025 09:28

Copilot AI reviewed Apr 4, 2025

View reviewed changes

htahir1 requested changes Apr 4, 2025

View reviewed changes

strickvl added 3 commits April 4, 2025 11:55

Fix CLI docs formatting

6fbaa1c

strickvl requested a review from htahir1 April 4, 2025 10:08

Merge branch 'develop' into doc/triggering-pipelines-and-ci-cd-tutorials

a279119

htahir1 requested changes Apr 4, 2025

View reviewed changes

strickvl added 2 commits April 4, 2025 14:06

Updates following review

76765dc

Merge branch 'doc/triggering-pipelines-and-ci-cd-tutorials' of github…

e2ac16d

….com:zenml-io/zenml into doc/triggering-pipelines-and-ci-cd-tutorials

Merge remote-tracking branch 'origin/develop' into doc/triggering-pip…

df582df

…elines-and-ci-cd-tutorials

strickvl marked this pull request as draft April 9, 2025 11:05

strickvl and others added 5 commits April 9, 2025 13:05

Merge branch 'develop' into doc/triggering-pipelines-and-ci-cd-tutorials

88fc981

Update Method 2 section headings to match Method 1 style

c3a4851

Revised heading structure in the tutorial to maintain consistent style between methods. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

add icon for page

3e9ce3a

Merge branch 'develop' into doc/triggering-pipelines-and-ci-cd-tutorials

af9f70b

strickvl requested review from htahir1 and Copilot April 9, 2025 15:29

strickvl marked this pull request as ready for review April 9, 2025 15:29

Copilot AI reviewed Apr 9, 2025

View reviewed changes

htahir1 requested changes Apr 9, 2025

View reviewed changes

docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md Show resolved Hide resolved

docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md Show resolved Hide resolved

strickvl and others added 2 commits April 9, 2025 17:39

Merge branch 'doc/triggering-pipelines-and-ci-cd-tutorials' of github…

9432a77

….com:zenml-io/zenml into doc/triggering-pipelines-and-ci-cd-tutorials

strickvl requested a review from htahir1 April 10, 2025 07:00

htahir1 requested changes Apr 10, 2025

View reviewed changes

htahir1 approved these changes Apr 10, 2025

View reviewed changes

strickvl merged commit 7ec8c6f into develop Apr 10, 2025
33 of 38 checks passed

strickvl deleted the doc/triggering-pipelines-and-ci-cd-tutorials branch April 10, 2025 10:01

Add triggering pipelines mini-tutorial #3495

Add triggering pipelines mini-tutorial #3495

Uh oh!

Conversation

strickvl commented Apr 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Apr 4, 2025

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

Uh oh!

strickvl commented Apr 4, 2025

Uh oh!

github-actions bot commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation Link Check Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htahir1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

strickvl commented Apr 9, 2025

Uh oh!

strickvl commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

htahir1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 4, 2025 •

edited

Loading