Start adding a KEP for csi volume resizing

keps/sig-storage/20190129-csi-volume-resizing.md

@@ -0,0 +1,170 @@
+---
+title: Support for CSI volume resizing
+authors:
+  - "@gnufied
+owning-sig: sig-storage
+participating-sigs:
+  - sig-storage
+reviewers:
+  - @saad-ali
+  - @jsafrane
+approvers:
+  - @saad-ali
+  - @childsb
+creation-date: 2019-01-29
+last-updated: 2019-01-29
+status: implementable
+see-also:
+    - [Kubernetes Volume expansion](https://github.com/kubernetes/enhancements/issues/284)
+    - [Online resizing design](https://github.com/kubernetes/enhancements/pull/737)
+replaces:
+superseded-by:
+---
+
+# Support for CSI volume resizing
+
+## Table of Contents
+
+Table of Contents
+=================
+
+   * [Support for CSI volume resizing](#support-for-csi-volume-resizing)
+      * [Table of Contents](#table-of-contents)
+   * [Table of Contents](#table-of-contents-1)
+      * [Summary](#summary)
+      * [Motivation](#motivation)
+         * [Goals](#goals)
+         * [Non-Goals](#non-goals)
+      * [Proposal](#proposal)
+         * [External resize controller](#external-resize-controller)
+         * [Expansion on Kubelet](#expansion-on-kubelet)
+             * [Offline volume resizing on kubelet:](#offline-volume-resizing-on-kubelet)
+             * [Online volume resizing on kubelet:](#online-volume-resizing-on-kubelet)
+         * [Risks and Mitigations](#risks-and-mitigations)
+      * [Test Plan](#test-plan)
+      * [Graduation Criteria](#graduation-criteria)
+      * [Implementation History](#implementation-history)
+
+
+## Summary
+
+To bring CSI volumes in feature parity with in-tree volumes we need to implement support for resizing of CSI volumes.
+
+## Motivation
+
+We recently implemented volume resizing support in CSI specs. This proposal implements this feature for Kubernetes.
+Any CSI volume plugin that implements necessary part of CSI specs will become resizable.
+
+### Goals
+
+To enable expansion of CSI volumes used by `PersistentVolumeClaim`s that support volume expansion as a plugin capability.
+
+### Non-Goals
+
+The expansion capability of a CSI plugin will not be validated by using CSI RPC call when user edits the PVC(i.e existing resize admission controller will not make CSI RPC call).
+The responsibility of
+actually enabling expansion for certains storageclasses still falls on Kubernetes admin.
+
+## Proposal
+
+The design of CSI volume resizing is made of two parts.
+
+
+### External resize controller
+
+To support resizing of CSI volumes an external resize controller will monitor all PVCs.  If a PVC meets following criteria for resizing, it will be added to
+controller's workqueue:
+
+- The driver name disovered from PVC should match name of driver currently known(by querying driver info via CSI RPC call) to external resize controller.
+- Once it notices a PVC has been updated and by comparing old and new PVC object, it determines more space has been requested by the user.
+
+Once PVC gets picked from workqueue, the controller will also compare requested PVC size with actual size of volume in `PersistentVolume`
+object. Once PVC passes all these checks, a CSI `ControllerExpandVolume` call will be made by the controller if CSI plugin implements `ControllerExpandVolume`
+RPC call.
+
+If `ControllerExpandVolume` call is successful and plugin implements `NodeExpandVolume`:
+- if `ControllerExpandVolumeResponse` returns `true` in `node_expansion_required` then `FileSystemResizePending` condition will be added to PVC and `NodeExpandVolume` operation will be queued on kubelet. Also volume size reported by PV will be updated to new value.
+- if `ControllerExpandVolumeResponse` returns `false` in `node_expansion_required` then volume resize operation will be marked finished and both `pvc.Status.Capacity` and `pv.Spec.Capacity` will report updated value.
+
+If plugin does not implement `NodeExpandVolume` then volume resize operation will be marked as finished and both `pvc.Status.Capacity` and `pv.Spec.Capacity` will report updated value after successful completion of `ControllerExpandVolume` RPC call.
+
+If `ControllerExpandVolume`  call fails:
+- Then PVC will retain `Resizing` condition and will have appropriate events added to the PVC.
+- Controller will retry resizing operation with exponential backoff, assuming it corrects itself.
+
+A general mechanism for recovering from resize failure will be implemented via: https://github.com/kubernetes/kubernetes/issues/73036
+
+### Expansion on Kubelet
+
+A CSI volume may require expansion on the node to finish volume resizing. In some cases - the entire resizing operation can happen on the node and
+plugin may choose to not implement `ControllerExpandVolume` CSI RPC call at all.
+
+Currently Kubernetes supports two modes of performing volume resize on kubelet. We will describe each mode here. For more information , please refer to original volume resize proposal - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/grow-volume-size.md.
+
+
+#### Offline volume resizing on kubelet:
+
+This is the default mode and in this mode `NodeExpandVolume` will only be called when volume is being mounted on the node. In other words, pod that was using the volume must be re-created for expansion on node to happen.
+
+When a pod that is using the PVC is started, kubelet will compare `pvc.spec.resources.requests.storage` and `pvc.Status.Capacity`. It also compares PVC's size with `pv.Spec.Capacity` and if it detects PV is reporting same size as pvc's spec but PVC's status is still reporting smaller value then it determines -
+a volume expansion is pending on the node.  At this point if plugin implements `NodeExpandVolume` RPC call then, kubelet will call it and:
+
+If `NodeExpandVolume` is successful:
+- It will update `pvc.Status.Capacity` with latest value and remove all resizing related conditions from PVC.
+
+If `NodeExpandVolume` failed:
+- It will add a event to both PVC and Pod about failed resizing and resize operation will be retried. This will prevent pod from starting up.
+
+
+#### Online volume resizing on kubelet:
+
+More details about online resizing can be found in [Online resizing design](https://github.com/kubernetes/enhancements/pull/737) but essentially if
+`ExpandInUsePersistentVolumes` feature is enabled then kubelet will periodically poll all PVCs that are being used on the node and compare `pvc.spec.resources.requests.storage` and `pvc.Status.Capacity`(also `pv.Spec.Capacity`) and make similar determination about whether node expansion is required for the volume.
+
+In this mode `NodeExpandVolume` can be called while pod is running and volume is in-use. Using aformentioned check if kubelet determines that
+volume expansion is needed on the node and plugin implements `NodeExpandVolume` RPC call then, kubelet will call it(provided volume has already been node staged and published on the node) and:
+
+If `NodeExpandVolume` is successful:
+- It will update `pvc.Status.Capacity` with latest value and remove all resizing related conditions from PVC.
+
+If `NodeExpandVolume` failed:
+- It will add a event to both PVC and Pod about failed resizing and resize operation will be retried.
+
+### Risks and Mitigations
+
+Before this feature goes GA - we need to handle recovering https://github.com/kubernetes/kubernetes/issues/73036.
+
+## Test Plan
+
+* Unit tests for external resize controller.
+* Add e2e tests in Kubernetes that use csi-mock driver for volume resizing.
+  - (postive) Give a plugin that supports both control plane and node size resize, CSI volume should be resizable and able to complete successfully.
+  - (positive) Given a plugin that only requires control plane resize, CSI volume should be resizable and able to complete successfully.
+  - (positive) Given a plugin that only requires node side resize, CSI volume should be resizable and able to complete successfully.
+  - (positive) Given a plugin that support online resizing, CSI volume should be resizable and online resize operation be able to complete successfully. 
+  - (negative) If control resize fails, PVC should have appropriate events.
+  - (neative) if node side resize fails, both pod and PVC should have appropriate events.
+
+## Graduation Criteria
+
+Once implemented CSI volumes should be resizable and in-line with current in-tree implementation of volume resizing.
+
+- *Alpha* : Initial support for CSI volume resizing. Released code will include an external CSI volume resize controller and changes to Kubelet. Implementation will have unit tests and csi-mock driver e2e tests.
+- *Beta*  : More robust support for CSI volume resizing, handle recovering from resize failures. Add e2e tests that use real drivers(`gce-pd`, `ebs` at minimum). Add metrics for volume resize operations.
+- *GA* : CSI resizing in general will only leave GA after existing [Volume expansion](https://github.com/kubernetes/enhancements/issues/284) feature leaves GA. Online resizing of CSI volumes depends on [Online resizing](https://github.com/kubernetes/enhancements/pull/737) feature and online resizing of CSI volumes will be available as a GA feature only when [Online resizing feature](https://github.com/kubernetes/enhancements/pull/737) goes GA.
+
+Hopefully the content previously contained in [umbrella issues][] will be tracked in the `Graduation Criteria` section.
+
+[umbrella issues]: https://github.com/kubernetes/kubernetes/issues/62096
+
+## Implementation History
+
+Major milestones in the life cycle of a KEP should be tracked in `Implementation History`.
+Major milestones might include
+
+- the `Summary` and `Motivation` sections being merged signaling SIG acceptance
+- the `Proposal` section being merged signaling agreement on a proposed design
+- the date implementation started
+- the first Kubernetes release where an initial version of the KEP was available
+- the version of Kubernetes where the KEP graduated to general availability
+- when the KEP was retired or superseded