TEP-0157 Retention Policy for Tekton Results

Author: khrm

teps/0157-retention-policy-results.md

@@ -0,0 +1,226 @@
+---
+status: proposed
+title: Retention Policy for Tekton Results
+creation-date: '2024-07-17'
+last-updated: '2024-07-17'
+authors:
+- '@khrm'
+collaborators: []
+---
+
+# TEP-0157: Tekton Results: Retention Policy for older Results and Records
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+  - [Use Cases](#use-cases)
+  - [Requirements](#requirements)
+- [Proposal](#proposal)
+  - [Notes and Caveats](#notes-and-caveats)
+- [Design Details](#design-details)
+- [Design Evaluation](#design-evaluation)
+  - [Reusability](#reusability)
+  - [Simplicity](#simplicity)
+  - [Flexibility](#flexibility)
+  - [User Experience](#user-experience)
+  - [Performance](#performance)
+  - [Risks and Mitigations](#risks-and-mitigations)
+  - [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+- [Implementation Plan](#implementation-plan)
+  - [Test Plan](#test-plan)
+  - [Infrastructure Needed](#infrastructure-needed)
+  - [Upgrade and Migration Strategy](#upgrade-and-migration-strategy)
+  - [Implementation Pull Requests](#implementation-pull-requests)
+- [References](#references)
+<!-- /toc -->
+
+## Summary
+Tekton Results stores Pipelineruns, TaskRuns, Events and Logs indefinitely.
+This proposed adding a retention policy feature for removing older Result and their associated records
+alongwith request to delete logs.
+
+## Motivation
+Storing older results and records indefinitely leads to wastage of storage resources
+and degradation of DB performance. Sometime we don't require some pipelines to be deleted from
+archives.
+
+### Goals
+- Ability to define retention period for the Results at cluster level. All records and results past that period should be deleted.
+- Ability to filter PipelineRuns when setting retention policy.
+- A way to delete associated logs also from s3 buckets, gcs buckets or PVC.
+
+### Non-Goals
+
+<!--
+Listing non-goals helps to focus discussion and make progress.
+- What is out of scope for this TEP?
+-->
+
+### Use Cases
+- User can specify a global policy for all the results. All records and logs falling under results satisfying pruning condition will be deleted.
+- User can filter results based on cel expression and result Summary expression. All the associated records will be deleted.
+
+### Requirements
+- For all results satisfying delete conditions, following things need to be deleted:
+* Results
+* Records for PipelineRun and TaskRun
+* Records for EventLog
+* Deletion of associated logs from s3 bucket, gcs bucket or PVC. EventLog Records should also be deleted.
+
+## Proposal
+A pruner will run which will spin up job at specified interval based on configmap `config-results-retention-policy` given ttl and cel expressions.
+
+### Notes and Caveats
+
+
+## Design Details
+Configmap will have following structure:
+`
+runAt: 5 4 * * 7 # Specify when to run job so that we can bring DB in maintenance mode if needed.
+max-retention: 2880h # Max Retention duration of 120 days.
+filters:
+- expr: summary.status == FAILURE # When there's failure in PipelineRun
+  ttl: 700h
+- expr: parent == dev         # When parent/namespace is dev
+  ttl: 300h
+`
+
+
+## Design Evaluation
+<!--
+How does this proposal affect the api conventions, reusability, simplicity, flexibility
+and conformance of Tekton, as described in [design principles](https://github.com/tektoncd/community/blob/master/design-principles.md)
+-->
+
+### Reusability
+
+<!--
+https://github.com/tektoncd/community/blob/main/design-principles.md#reusability
+
+- Are there existing features related to the proposed features? Were the existing features reused?
+- Is the problem being solved an authoring-time or runtime-concern? Is the proposed feature at the appropriate level
+authoring or runtime?
+-->
+
+### Simplicity
+
+<!--
+https://github.com/tektoncd/community/blob/main/design-principles.md#simplicity
+
+- How does this proposal affect the user experience?
+- What’s the current user experience without the feature and how challenging is it?
+- What will be the user experience with the feature? How would it have changed?
+- Does this proposal contain the bare minimum change needed to solve for the use cases?
+- Are there any implicit behaviors in the proposal? Would users expect these implicit behaviors or would they be
+surprising? Are there security implications for these implicit behaviors?
+-->
+
+### Flexibility
+
+<!--
+https://github.com/tektoncd/community/blob/main/design-principles.md#flexibility
+
+- Are there dependencies that need to be pulled in for this proposal to work? What support or maintenance would be
+required for these dependencies?
+- Are we coupling two or more Tekton projects in this proposal (e.g. coupling Pipelines to Chains)?
+- Are we coupling Tekton and other projects (e.g. Knative, Sigstore) in this proposal?
+- What is the impact of the coupling to operators e.g. maintenance & end-to-end testing?
+- Are there opinionated choices being made in this proposal? If so, are they necessary and can users extend it with
+their own choices?
+-->
+
+### Conformance
+
+<!--
+https://github.com/tektoncd/community/blob/main/design-principles.md#conformance
+
+- Does this proposal require the user to understand how the Tekton API is implemented?
+- Does this proposal introduce additional Kubernetes concepts into the API? If so, is this necessary?
+- If the API is changing as a result of this proposal, what updates are needed to the
+[API spec](https://github.com/tektoncd/pipeline/blob/main/docs/api-spec.md)?
+-->
+
+### User Experience
+
+<!--
+(optional)
+
+Consideration about the user experience. Depending on the area of change,
+users may be Task and Pipeline editors, they may trigger TaskRuns and
+PipelineRuns or they may be responsible for monitoring the execution of runs,
+via CLI, dashboard or a monitoring system.
+
+Consider including folks that also work on CLI and dashboard.
+-->
+
+### Performance
+This improves the peformance of DB by deleting superfluous results and their associated datas.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Tekton ecosystem. Consider including folks that also work outside the WGs
+or subproject.
+- How will security be reviewed and by whom?
+- How will UX be reviewed and by whom?
+-->
+
+### Drawbacks
+
+<!--
+Why should this TEP _not_ be implemented?
+-->
+
+## Alternatives
+
+
+## Implementation Plan
+
+<!--
+What are the implementation phases or milestones? Taking an incremental approach
+makes it easier to review and merge the implementation pull request.
+-->
+
+
+### Test Plan
+
+- We will add a Integration tests like we have for Logging in GCS storage and other scenarios.
+
+### Infrastructure Needed
+
+<!--
+(optional)
+
+Use this section if you need things from the project or working group.
+Examples include a new subproject, repos requested, GitHub details.
+Listing these here allows a working group to get the process for these
+resources started right away.
+-->
+
+### Upgrade and Migration Strategy
+
+<!--
+(optional)
+
+Use this section to detail whether this feature needs an upgrade or
+migration strategy. This is especially useful when we modify a
+behavior or add a feature that may replace and deprecate a current one.
+-->
+
+### Implementation Pull Requests
+
+
+## References
+
+<!--
+(optional)
+
+Use this section to add links to GitHub issues, other TEPs, design docs in Tekton
+shared drive, examples, etc. This is useful to refer back to any other related links
+to get more details.
+-->