Skip to content

Better input artifacts typing #3099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 67 commits into from
Nov 7, 2024
Merged

Better input artifacts typing #3099

merged 67 commits into from
Nov 7, 2024

Conversation

avishniakov
Copy link
Contributor

@avishniakov avishniakov commented Oct 18, 2024 •
edited
Loading

Describe changes

I implemented helpers to define what is the actual input artifact type in a Step. It can be:

  • STEP_OUTPUT for regular case
  • EXTERNAL for ExternalArtifact case
  • MANUAL for load_artifact or Client().get_artifact_version inside a step
  • LAZY_LOADED for lazy load cases

StepRunResponse is now equipped with input_types for DAG.

I also extended Client().get_artifact_version to link the artifact back to the step as input of MANUAL type.

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
  • If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

avishniakov and others added 25 commits October 10, 2024 17:21
@avishniakov
multi-versioned outputs
9bdb5f1
@avishniakov
more artifact save types
4dd7e90
@avishniakov
fix for the llm template
eb7f691
@actions-user
Auto-update of LLM Finetuning template
49f0f24
@avishniakov
Merge branch 'develop' into feature/PRD-663-multiple-output-versions-…
5d81b5d 
…for-a-step
@actions-user
Auto-update of Starter template
ac9a7bd
@actions-user
Auto-update of E2E template
9607df5
@avishniakov
fix tests notation
fc19dc3
@avishniakov
Merge branch 'feature/PRD-663-multiple-output-versions-for-a-step' of h…
9f04da1 
…ttps://github.com/zenml-io/zenml into feature/PRD-663-multiple-output-versions-for-a-step
@avishniakov
Refactor artifact saving logic to use save types
feb3f53
@avishniakov
Refactor artifact saving logic to use save types
5557837
@avishniakov
Remove unneeded TODO
eac4aea
@avishniakov
Refactor artifact saving logic to use list instead of set for output …
e62d8eb 
…artifacts
@avishniakov
Merge branch 'develop' into feature/PRD-663-multiple-output-versions-…
f4bf134 
…for-a-step
@avishniakov
Refactor artifact saving logic to use outputs instead of saved_artifa…
9a8cf20 
…ct_versions
@avishniakov
Merge branch 'develop' into feature/PRD-663-multiple-output-versions-…
ecb9ff0 
…for-a-step
@actions-user
Auto-update of LLM Finetuning template
5683081
@avishniakov
Refactor artifact saving logic to use 'external' as the default save …
f02fee1 
…type
@avishniakov
Refactor artifact saving logic to use outputs instead of saved_artifa…
0a2dfa1 
…ct_versions
@avishniakov
Merge branch 'develop' into feature/PRD-663-multiple-output-versions-…
77b6a79 
…for-a-step
@avishniakov
Refactor artifact saving logic to define input types. WIP
27099b4
@avishniakov
Refactor StepRunRequestFactory to correctly assign output artifacts
a704f95
@avishniakov
Merge branch 'feature/PRD-663-multiple-output-versions-for-a-step' of h…
e991fd5 
…ttps://github.com/zenml-io/zenml into feature/PRD-663-multiple-output-versions-for-a-step
@avishniakov
Merge branch 'feature/PRD-663-multiple-output-versions-for-a-step' in…
e7d7a62 
…to feature/PRD-668-better-input-artifacts-typing
@avishniakov
Refactor StepRunRequestFactory to correctly assign output artifacts
9ce83ee
@avishniakov avishniakov requested a review from schustmi October 18, 2024 09:49
@avishniakov avishniakov changed the base branch from main to develop October 18, 2024 09:49
@github-actions github-actions bot added internal To filter out internal PRs and issues enhancement New feature or request labels Oct 18, 2024
@avishniakov avishniakov changed the base branch from develop to feature/PRD-663-multiple-output-versions-for-a-step October 18, 2024 09:49
schustmi and others added 12 commits October 30, 2024 14:04
@schustmi
Detect input type server side
ce76a9c
@schustmi
Fix input type detection
8b51d47
@schustmi
Don't fail for cached step runs
5f1c7fc
@schustmi
Merge branch 'develop' into feature/PRD-663-multiple-output-versions-…
1ecefd8 
…for-a-step
@schustmi
Fix alembic order
9a39cd4
@schustmi
Merge branch 'feature/PRD-663-multiple-output-versions-for-a-step' in…
48b28f2 
…to feature/PRD-668-better-input-artifacts-typing
@avishniakov
Refactor artifact saving in cacheable_multiple_versioned_producer
3193c20 
This commit refactors the artifact saving logic in the `cacheable_multiple_versioned_producer` function. It introduces two new parameters, `is_model_artifact` and `is_deployment_artifact`, which allow specifying the type of the saved artifact. This change improves the flexibility and customization of the function.

Related to #PRD-663
@avishniakov
Merge branch 'develop' into feature/PRD-663-multiple-output-versions-…
3797a07 
…for-a-step
@avishniakov
Merge branch 'feature/PRD-663-multiple-output-versions-for-a-step' in…
e49ff02 
…to feature/PRD-668-better-input-artifacts-typing
@avishniakov
remove duplicated logic from load_artifact
2986c7a
@avishniakov
add StepRunInputResponse
5b6bf57
@avishniakov
add StepRunInputResponse
f9c4455
@avishniakov avishniakov requested a review from schustmi November 6, 2024 09:38
Base automatically changed from feature/PRD-663-multiple-output-versions-for-a-step to develop November 7, 2024 06:36
@avishniakov avishniakov merged commit a37af63 into develop Nov 7, 2024
67 of 68 checks passed
@avishniakov avishniakov deleted the feature/PRD-668-better-input-artifacts-typing branch November 7, 2024 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@schustmi schustmi schustmi approved these changes

@bcdurak bcdurak Awaiting requested review from bcdurak

Assignees
No one assigned
Labels
enhancement New feature or request internal To filter out internal PRs and issues run-slow-ci
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@avishniakov @schustmi @actions-user