You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
make sure that a reconciliation happens when a custom resource is instructed to be deleted. Typical case when it's
39
39
useful, when an operator is down (pod not running). Without a finalizer the reconciliation - thus the cleanup
40
-
i.e. [`ResourceController.deleteResource(...)`](https://github.com/java-operator-sdk/java-operator-sdk/blob/master/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/ResourceController.java)
40
+
i.e. [`Reconciler.cleanup(...)`](https://github.com/java-operator-sdk/java-operator-sdk/blob/master/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/ResourceController.java)
41
41
42
42
- would not happen if a custom resource is deleted.
43
43
44
44
Finalizers are automatically added by the framework as the first step, thus when a custom resource is created, but
45
-
before the first reconciliation, the custom resource is updated via a Kubernetes API call. As a result of this update, the
46
-
finalizer will be present. The subsequent event will be received, which will trigger the first reconciliation.
45
+
before the first reconciliation, the custom resource is updated via a Kubernetes API call. As a result of this update,
46
+
the finalizer will be present. The subsequent event will be received, which will trigger the first reconciliation.
47
47
48
-
The finalizer that is automatically added will be also removed after the `deleteResource` is executed on the controllerConfiguration.
49
-
However, the removal behavior can be further customized, and can be instructed to "not remove yet" - this is useful just
50
-
in some specific corner cases, when there would be a long waiting period for some dependent resource cleanup.
48
+
The finalizer that is automatically added will be also removed after the `deleteResource` is executed on the
49
+
controllerConfiguration. However, the removal behavior can be further customized, and can be instructed to "not remove
50
+
yet" - this is useful just in some specific corner cases, when there would be a long waiting period for some dependent
51
+
resource cleanup.
51
52
52
53
The name of the finalizers can be specified, in case it is not, a name will be generated.
53
54
@@ -67,95 +68,140 @@ When automatic finalizer handling is turned off, the `ResourceController.deleteR
67
68
case of a delete event received. So it does not make sense to implement this method and turn off finalizer at the same
68
69
time.
69
70
70
-
## The `createOrUpdateResource` and `deleteResource` Methods of `ResourceController`
71
+
## The `reconcile` and `cleanup` Methods of `Reconciler`
71
72
72
-
The lifecycle of a custom resource can be clearly separated to two phases from a perspective of an operator.
73
-
When a custom resource is created or update, or on the other hand when the custom resource is deleted - or rater
74
-
marked for deletion in case a finalizer is used.
73
+
The lifecycle of a custom resource can be clearly separated to two phases from a perspective of an operator. When a
74
+
custom resource is created or update, or on the other hand when the custom resource is deleted - or rater marked for
75
+
deletion in case a finalizer is used.
75
76
76
-
There is no point to make a distinction between create and update, since the reconciliation
77
-
logic typically would be very similar or identical in most of the cases.
77
+
There is no point to make a distinction between create and update, since the reconciliation logic typically would be
78
+
very similar or identical in most of the cases.
78
79
79
-
This separation related logic is automatically handled by framework. The framework will always call `createOrUpdateResource`
80
-
function, unless the custom resource is
81
-
[marked from deletion](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/#how-finalizers-work).
82
-
From the point when the custom resource is marked from deletion, only the `deleteResource` method is called.
80
+
This separation related logic is automatically handled by framework. The framework will always
81
+
call `createOrUpdateResource`
82
+
function, unless the custom resource is
83
+
[marked from deletion](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/#how-finalizers-work)
84
+
. From the point when the custom resource is marked from deletion, only the `deleteResource` method is called.
83
85
84
86
If there is **no finalizer** in place (see Finalizer Support section), the `deleteResource` method is **not called**.
85
87
86
88
### Using `UpdateControl` and `DeleteControl`
87
89
88
-
These two methods are used to control the outcome or the desired behavior after the reconciliation.
90
+
These two methods are used to control the outcome or the desired behavior after the reconciliation.
89
91
90
-
The `UpdateControl` can instruct the framework to update the custom resource status sub-resource and/or re-schedule
91
-
a reconciliation with a desired time delay. Those are the typical use cases, however in some cases there it can happen
92
-
that the controller wants to update the custom resource itself (like adding annotations) or not to do any updates,
93
-
which are also supported.
92
+
The `UpdateControl` can instruct the framework to update the custom resource status sub-resource and/or re-schedule a
93
+
reconciliation with a desired time delay. Those are the typical use cases, however in some cases there it can happen
94
+
that the controller wants to update the custom resource itself (like adding annotations) or not to do any updates, which
95
+
are also supported.
94
96
95
-
It is also possible to update both the status and the custom resource with `updateCustomResourceAndStatus` method.
96
-
In this case first the custom resource is updated then the status in two separate requests to K8S API.
97
+
It is also possible to update both the status and the custom resource with `updateCustomResourceAndStatus` method. In
98
+
this case first the custom resource is updated then the status in two separate requests to K8S API.
97
99
98
100
Always update the custom resource with `UpdateControl`, not with the actual kubernetes client if possible.
99
101
100
102
On custom resource updates there is always an optimistic version control in place, to make sure that another update is
101
-
not overwritten (by setting `resourceVersion` ) .
103
+
not overwritten (by setting `resourceVersion` ) .
102
104
103
-
The `DeleteControl` typically instructs the framework to remove the finalizer after the dependent resource are
104
-
cleaned up in `deleteResource` implementation.
105
+
The `DeleteControl` typically instructs the framework to remove the finalizer after the dependent resource are cleaned
106
+
up in `deleteResource` implementation.
105
107
106
-
However, there is a possibility to not remove the finalizer, this
107
-
allows to clean up the resources in a more async way, mostly for the cases when there is a long waiting period after a delete
108
-
operation is initiated. Note that in this case you might want to either schedule a timed event to make sure the
109
-
`deleteResource` is executed again or use event sources get notified about the state changes of a deleted resource.
108
+
However, there is a possibility to not remove the finalizer, this allows to clean up the resources in a more async way,
109
+
mostly for the cases when there is a long waiting period after a delete operation is initiated. Note that in this case
110
+
you might want to either schedule a timed event to make sure the
111
+
`deleteResource` is executed again or use event sources get notified about the state changes of a deleted resource.
110
112
111
113
## Automatic Retries on Error
112
114
113
-
When an exception is thrown from a controller, the framework will schedule an automatic retry of the reconciliation.
114
-
The retry is behavior is configurable, an implementation is provided that should cover most of the use-cases, see
115
+
When an exception is thrown from a controller, the framework will schedule an automatic retry of the reconciliation. The
116
+
retry is behavior is configurable, an implementation is provided that should cover most of the use-cases, see
But it is possible to provide a custom implementation.
117
119
118
-
It is possible to set a limit on the number of retries. In the [Context](https://github.com/java-operator-sdk/java-operator-sdk/blob/master/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/Context.java)
119
-
object information is provided about the retry, particularly interesting is the `isLastAttempt`, since a behavior
120
-
could be implemented bases on this flag. Like setting an error message in the status in case of a last attempt;
120
+
It is possible to set a limit on the number of retries. In
121
+
the [Context](https://github.com/java-operator-sdk/java-operator-sdk/blob/master/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/Context.java)
122
+
object information is provided about the retry, particularly interesting is the `isLastAttempt`, since a behavior could
123
+
be implemented bases on this flag. Like setting an error message in the status in case of a last attempt;
121
124
122
-
Event if the retry reached a limit, in case of a new event is received the reconciliation would happen again, it's
123
-
just won't be a result of a retry, but the new event.
125
+
Event if the retry reached a limit, in case of a new event is received the reconciliation would happen again, it's just
126
+
won't be a result of a retry, but the new event.
124
127
125
128
A successful execution resets the retry.
126
129
127
130
### Correctness and Automatic Retries
128
131
129
-
There is a possibility to turn of the automatic retries. This is not desirable, unless there is a very specific
130
-
reason. Errors naturally happen, typically network errors can cause some temporal issues, another case is when a
131
-
custom resource is updated during the reconciliation (using `kubectl` for example), in this case
132
-
if an update of the custom resource from the controller (using `UpdateControl`) would fail on a conflict. The automatic
133
-
retries covers these cases and will result in a reconciliation, even if normally an event would not be processed
134
-
as a result of a custom resource update from previous example (like if there is no generation update as a result of the
135
-
change and generation filtering is turned on)
132
+
There is a possibility to turn of the automatic retries. This is not desirable, unless there is a very specific reason.
133
+
Errors naturally happen, typically network errors can cause some temporal issues, another case is when a custom resource
134
+
is updated during the reconciliation (using `kubectl` for example), in this case if an update of the custom resource
135
+
from the controller (using `UpdateControl`) would fail on a conflict. The automatic retries covers these cases and will
136
+
result in a reconciliation, even if normally an event would not be processed as a result of a custom resource update
137
+
from previous example (like if there is no generation update as a result of the change and generation filtering is
138
+
turned on)
136
139
137
-
## Re-Scheduling Execution
140
+
## Rescheduling Execution
138
141
139
-
In simple operators one way to implement an operator is to periodically reconcile it. This is supported explicitly by
140
-
`UpdateControl`, see method: `public UpdateControl<T> withReSchedule(long delay, TimeUnit timeUnit)`.
141
-
This would schedule a reconciliation to the future.
142
+
One way to implement an operator especially in simple cases is to periodically reconcile it. This is supported
143
+
explicitly by
144
+
`UpdateControl`, see method: `public UpdateControl<T> rescheduleAfter(long delay, TimeUnit timeUnit)`. This would
145
+
schedule a reconciliation for the future.
142
146
143
-
## Retry and Re-Scheduling Common Behavior
147
+
## Retry and Rescheduling and Unrelated Event Handling Common Behavior
148
+
149
+
Retry, reschedule and standard event processing forms a relatively complex system, where these functionalities are not
150
+
independent of each other. In the following we describe the behavior in this section, so it is easier to understand the
151
+
intersections:
152
+
153
+
1. A successful execution resets a retry and the rescheduled executions which were present before the reconciliation.
154
+
However, a new rescheduling can be instructed from the reconciliation outcome (`UpdateControl` or `DeleteControl`).
155
+
2. In case an exception happened, and a retry is initiated, but an event received meanwhile the reconciliation will be
156
+
executed instantly, and this execution won't count as a retry attempt.
157
+
3. If the retry limit is reached (so no more automatic retry would happen), but a new event received, the reconciliation
158
+
will still happen, but won't reset the retry, will be still marked as the last attempt in the retry info. The point
159
+
1. holds, but in case of an error, no retry will happen.
144
160
145
161
## Handling Related Events with Event Sources
146
162
163
+
See also this [blog post](https://csviri.medium.com/java-operator-sdk-introduction-to-event-sources-a1aab5af4b7b).
164
+
165
+
Event sources are a relatively simple yet powerful and extensible concept to trigger controller executions. Usually
166
+
based on changes of dependent resources. To solve the mentioned problems above, de-facto we watch resources we manage
167
+
for changes, and reconcile the state if a resource is changed. Note that resources we are watching can be Kubernetes and
168
+
also non-Kubernetes objects. Typically, in case of non-Kubernetes objects or services we can extend our operator to
169
+
handle webhooks or websockets or to react to any event coming from a service we interact with. What happens is when we
170
+
create a dependent resource we also register an Event Source that will propagate events regarding the changes of that
171
+
resource. This way we avoid the need of polling, and can implement controllers very efficiently.
172
+
173
+
174
+
175
+
There are few interesting points here:
176
+
The CustomResourceEvenSource event source is a special one, which sends events regarding changes of our custom resource,
177
+
this is an event source which is always registered for every controller by default. An event is always related to a
178
+
custom resource. Concurrency is still handled for you, thus we still guarantee that there is no concurrent execution of
179
+
the controller for the same custom resource (
180
+
there is parallel execution if an event is related to another custom resource instance).
181
+
147
182
### Caching and Event Sources
148
183
149
-
### The CustomResourceEventSource
184
+
Typically, when we work with Kubernetes (but possibly with others), we manage the objects in a declarative way. This is
185
+
true also for Event Sources. For example if we watch for changes of a Kubernetes Deployment object in the
186
+
InformerEventSource, we always receive the whole object from the Kubernetes API. Later when we try to reconcile in the
187
+
controller (not using events) we would like to check the state of this deployment (but also other dependent resources),
188
+
we could read the object again from Kubernetes API. However since we watch for the changes we know that we always
189
+
receive the most up-to-date version in the Event Source. So naturally, what we can do is cache the latest received
190
+
objects (in the Event Source) and read it from there if needed.
150
191
151
192
### Built-in Event Sources
152
193
194
+
1. InformerEventSource - used to get event about other K8S resources, also provides a local cache for them.
195
+
2. TimerEventSource - used to create timed events, mainly intended for internal usage.
196
+
3. CustomResourceEventSource - an event source that is automatically registered to listen to the changes of the main
197
+
resource the operation manages, it also maintains a cache of those objects that can be accessed from the Reconciler.
198
+
153
199
## Monitoring with Micrometer
154
200
155
201
## Contextual Info for Logging with MDC
156
202
157
-
Logging is enhanced with additional contextual information using [MDC](http://www.slf4j.org/manual.html#mdc).
158
-
This following attributes are available in most parts of reconciliation logic and during the execution of the controller:
203
+
Logging is enhanced with additional contextual information using [MDC](http://www.slf4j.org/manual.html#mdc). This
204
+
following attributes are available in most parts of reconciliation logic and during the execution of the controller:
0 commit comments