Skip to content

Commit 3e00963

Browse files
bogdandrutubogdandrutu
authored
Propose sampling API changes (open-telemetry#24)
* Propose sampling API changes * Add details about SamplingHint, Sampler interface and default samplers. * Add details about start span operation. * Update spelling text/0006-sampling.md Co-Authored-By: Yang Song <[email protected]> * Update spelling text/0006-sampling.md Co-Authored-By: Yang Song <[email protected]> * Add span name to the Sampler interface. Clarify how the sampling flags are exposed. * Use ascii for personas * Add spankind to the sampler input. * Remove delayed span creation proposal from this RFC * Remove unspecified type from sampler hint. * Update text/0006-sampling.md Co-Authored-By: Tristan Sloughter <[email protected]> * Add clarification that Sampler is always called.
1 parent f94eb1e commit 3e00963

File tree

1 file changed

+337
-0
lines changed

1 file changed

+337
-0
lines changed

‎0006-sampling.md

Lines changed: 337 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,337 @@
1+
# Sampling API
2+
3+
*Status: proposed*
4+
5+
## TL;DR
6+
This section tries to summarize all the changes proposed in this RFC:
7+
1. Move the `Sampler` interface from the API to SDK package. Apply some minor changes to the
8+
`Sampler` API.
9+
1. Add a new `SamplerHint` concept to the API package.
10+
1. Add capability to record `Attributes` that can be used for sampling decision during the `Span`
11+
creation time.
12+
1. Remove `addLink` APIs from the `Span` interface, and allow recording links only during the span
13+
construction time.
14+
15+
## Motivation
16+
17+
Different users of OpenTelemetry, ranging from library developers, packaged infrastructure binary
18+
developers, application developers, operators, and telemetry system owners, have separate use cases
19+
for OpenTelemetry that have gotten muddled in the design of the original Sampling API. Thus, we need
20+
to clarify what APIs each should be able to depend upon, and how they will configure sampling and
21+
OpenTelemetry according to their needs.
22+
23+
```
24+
25+
+----------+ +-----------+
26+
grpc | Library | | |
27+
Django | Devs +---------->| OTel API |
28+
Express | | +------>| |
29+
+----------+ | +--->+-----------+ +---------+
30+
| | ^ | OTel |
31+
| | | +->| Proxy +---+
32+
| | | | | | |
33+
+----------+ | | +-----+-----+------------+ | +---------+ |
34+
| | | | | | OTel Wire | | |
35+
Hbase | Infra | | | | | Export |+-+ v
36+
Envoy | Binary +---+ | | OTel | | | +----v-----+
37+
| Devs | | | SDK +------------+ | | |
38+
+----------+---------->| | | +---------->| Backend |
39+
+------>| | Custom | +---------->| |
40+
| | | | Export | | +----------+
41+
+----------+ | | | | |+-+ ^
42+
| +---+ | +-----------+------------+ |
43+
| App +------+ ^ ^ |
44+
| Devs + | | +------------+-+
45+
| | | | | |
46+
+----------+ +---+----+ +----------+ Telemetry |
47+
| SRE | | Owner |
48+
| | | |
49+
+--------+ +--------------+
50+
Lightstep
51+
Honeycomb
52+
53+
```
54+
## Explanation
55+
56+
We outline five different use cases (who may be overlapping sets of people), and how they should
57+
interact with OpenTelemetry:
58+
59+
### Library developer
60+
Examples: gRPC, Express, Django developers.
61+
62+
* They must only depend upon the OpenTelemetry API and not upon the SDK.
63+
* For testing only they may depend on the SDK with InMemoryExporter.
64+
* They are shipping source code that will be linked into others' applications.
65+
* They have no explicit runtime control over the application.
66+
* They know some signal about what traces may be interesting (e.g. unusual control plane requests)
67+
or uninteresting (e.g. health-checks), but have to write fully generically.
68+
69+
**Solution:**
70+
71+
* On the start Span operation, the OpenTelemetry API will allow marking a span with one of three
72+
choices for the [SamplingHint](#samplinghint).
73+
74+
### Infrastructure package/binary developer
75+
Examples: HBase, Envoy developers.
76+
77+
* They are shipping self-contained binaries that may accept YAML or similar run-time configuration,
78+
but are not expected to support extensibility/plugins beyond the default OpenTelemetry SDK,
79+
OpenTelemetry SDKTracer, and OpenTelemetry wire format exporter.
80+
* They may have their own recommendations for sampling rates, but don't run the binaries in
81+
production, only provide packaged binaries. So their sampling rate configs, and sampling strategies
82+
need to a finite "built in" set from OpenTelemetry's SDK.
83+
* They need to deal with upstream sampling decisions made by services calling them.
84+
85+
**Solution:**
86+
* Allow different sampling strategies by default in OpenTelemetry SDK, all configurable easily via
87+
YAML or feature flags. See [default samplers](#default-samplers).
88+
89+
### Application developer
90+
These are the folks we've been thinking the most about for OpenTelemetry in general.
91+
92+
* They have full control over the OpenTelemetry implementation or SDK configuration. When using the
93+
SDK they can configure custom exporters, custom code/samplers, etc.
94+
* They can choose to implement runtime configuration via a variety of means (e.g. baking in feature
95+
flags, reading YAML files, etc.), or even configure the library in code.
96+
* They make heavy usage of OpenTelemetry for instrumenting application-specific behavior, beyond
97+
what may be provided by the libraries they use such as gRPC, Django, etc.
98+
99+
**Solution:**
100+
* Allow application developers to link in custom samplers or write their own when using the
101+
official SDK.
102+
* These might include dynamic per-field sampling to achieve a target rate
103+
(e.g. https://github.com/honeycombio/dynsampler-go)
104+
* Sampling decisions are made within the start Span operation, after attributes relevant to the
105+
span have been added to the Span start operation but before a concrete Span object exists (so that
106+
either a NoOpSpan can be made, or an actual Span instance can be produced depending upon the
107+
sampler's decision).
108+
* Span.IsRecording() needs to be present to allow costly span attribute/log computation to be
109+
skipped if the span is a NoOp span.
110+
111+
### Application operator
112+
Often the same people as the application developers, but not necessarily
113+
114+
* They care about adjusting sampling rates and strategies to meet operational needs, debugging,
115+
and cost.
116+
117+
**Solution:**
118+
* Use config files or feature flags written by the application developers to control the
119+
application sampling logic.
120+
* Use the config files to configure libraries and infrastructure package behavior.
121+
122+
### Telemetry infrastructure owner
123+
They are the people who provide an implementation for the OpenTelemetry API by using the SDK with
124+
custom `Exporter`s, `Sampler`s, hooks, etc. or by writing a custom implementation, as well as
125+
running the infrastructure for collecting exported traces.
126+
127+
* They care about a variety of things, including efficiency, cost effectiveness, and being able to
128+
gather spans in a way that makes sense for them.
129+
130+
**Solution:**
131+
* Infrastructure owners receive information attached to the span, after sampling hooks have already
132+
been run.
133+
134+
## Internal details
135+
In Dapper based systems (or systems without a deferred sampling decision) all exported spans are
136+
stored to the backend, thus some of these systems usually don't scale to a high volume of traces,
137+
or the cost to store all the Spans may be too high. In order to support this use-case and to
138+
ensure the quality of the data we send, OpenTelemetry needs to natively support sampling with some
139+
requirements:
140+
* Send as many complete traces as possible. Sending just a subset of the spans from a trace is
141+
less useful because in this case the interaction between the spans may miss.
142+
* Allow application operator to configure the sampling frequency.
143+
144+
For new modern systems that need to collect all the Spans and later may or may not do a deferred
145+
sampling decision, OpenTelemetry needs to natively support a way to configure the library to
146+
collect and export all the Spans. This is possible even though OpenTelemetry supports sampling by
147+
setting a default config to always collect all the spans.
148+
149+
### Sampling flags
150+
OpenTelemetry API has two flags/properties:
151+
* `RecordEvents`
152+
* This property is exposed in the `Span` interface (e.g. `Span.isRecordingEvents()`).
153+
* If `true` the current `Span` records tracing events (attributes, events, status, etc.),
154+
otherwise all tracing events are dropped.
155+
* Users can use this property to determine if expensive trace events can be avoided.
156+
* `SampledFlag`
157+
* This flag is propagated via the `TraceOptions` to the child Spans (e.g.
158+
`TraceOptions.isSampled()`). For more details see the w3c definition [here][trace-flags].
159+
* In Dapper based systems this is equivalent to `Span` being `sampled` and exported.
160+
161+
The flag combination `SampledFlag == false` and `RecordEvents == true` means that the current `Span`
162+
does record tracing events, but most likely the child `Span` will not. This combination is
163+
necessary because:
164+
* Allow users to control recording for individual Spans.
165+
* OpenCensus has this to support z-pages, so we need to keep backwards compatibility.
166+
167+
The flag combination `SampledFlag == true` and `RecordEvents == false` can cause gaps in the
168+
distributed trace, and because of this OpenTelemetry API should NOT allow this combination.
169+
170+
It is safe to assume that users of the API should only access the `RecordEvents` property when
171+
instrumenting code and never access `SampledFlag` unless used in context propagators.
172+
173+
### SamplingHint
174+
This is a new concept added in the OpenTelemetry API that allows to suggest sampling hints to the
175+
implementation of the API:
176+
* `NOT_RECORD`
177+
* Suggest to not `RecordEvents = false` and not propagate `SampledFlag = false`.
178+
* `RECORD`
179+
* Suggest `RecordEvents = true` and `SampledFlag = false`.
180+
* `RECORD_AND_PROPAGATE`
181+
* Suggest to `RecordEvents = true` and propagate `SampledFlag = true`.
182+
183+
The default option for the span creation is to not have any suggestion (or suggestion is not
184+
specified). This can be implemented by using `null` as the default option or any language specific
185+
mechanism to achieve the same result.
186+
187+
### Sampler interface
188+
The interface for the Sampler class that is available only in the OpenTelemetry SDK:
189+
* `TraceID`
190+
* `SpanID`
191+
* Parent `SpanContext` if any
192+
* `SamplerHint`
193+
* `Links`
194+
* Span name
195+
* `SpanKind`
196+
* Initial set of `Attributes` for the `Span` being constructed
197+
198+
It produces as an output called `SamplingResult`:
199+
* A `SamplingDecision` enum [`NOT_RECORD`, `RECORD`, `RECORD_AND_PROPAGATE`].
200+
* A set of span Attributes that will also be added to the `Span`.
201+
* These attributes will be added after the initial set of `Attributes`.
202+
* (under discussion in separate RFC) the SamplingRate float.
203+
204+
### Default Samplers
205+
These are the default samplers implemented in the OpenTelemetry SDK:
206+
* ALWAYS_ON
207+
* Ignores all values in SamplingHint.
208+
* ALWAYS_OFF
209+
* Ignores all values in SamplingHint.
210+
* ALWAYS_PARENT
211+
* Ignores all values in SamplingHint.
212+
* Trust parent sampling decision (trusting and propagating parent `SampledFlag`).
213+
* For root Spans (no parent available) returns `NOT_RECORD`.
214+
* Probability
215+
* Allows users to configure to ignore or not the SamplingHint for every value different than
216+
`UNSPECIFIED`.
217+
* Default is to NOT ignore `NOT_RECORD` and `RECORD_AND_PROPAGATE` but ignores `RECORD`.
218+
* Allows users to configure to ignore the parent `SampledFlag`.
219+
* Allows users to configure if probability applies only for "root spans", "root spans and remote
220+
parent", or "all spans".
221+
* Default is to apply only for "root spans and remote parent".
222+
* Remote parent property should be added to the SpanContext see specs [PR/216][specs-pr-216]
223+
* Sample with 1/N probability
224+
225+
**Root Span Decision:**
226+
227+
|Sampler|RecordEvents|SampledFlag|
228+
|---|---|---|
229+
|ALWAYS_ON|`True`|`True`|
230+
|ALWAYS_OFF|`False`|`False`|
231+
|ALWAYS_PARENT|`False`|`False`|
232+
|Probability|`SamplingHint==RECORD OR SampledFlag`|`SamplingHint==RECORD_AND_PROPAGATE OR Probability`|
233+
234+
**Child Span Decision:**
235+
236+
|Sampler|RecordEvents|SampledFlag|
237+
|---|---|---|
238+
|ALWAYS_ON|`True`|`True`|
239+
|ALWAYS_OFF|`False`|`False`|
240+
|ALWAYS_PARENT|`ParentSampledFlag`|`ParentSampledFlag`|
241+
|Probability|`SamplingHint==RECORD OR SampledFlag`|`ParentSampledFlag OR SamplingHint==RECORD_AND_PROPAGATE OR Probability`|
242+
243+
### Links
244+
This RFC proposes that Links will be recorded only during the start `Span` operation, because:
245+
* Link's `SampledFlag` can be used in the sampling decision.
246+
* OpenTracing supports adding references only during the `Span` creation.
247+
* OpenCensus supports adding links at any moment, but this was mostly used to record child Links
248+
which are not supported in OpenTelemetry.
249+
* Allowing links to be recorded after the sampling decision is made will cause samplers to not
250+
work correctly and unexpected behaviors for sampling.
251+
252+
### When does sampling happen?
253+
The sampling decision will happen before a real `Span` object is returned to the user, because:
254+
* If child spans are created they need to know the 'SampledFlag'.
255+
* If `SpanContext` is propagated on the wire the 'SampledFlag' needs to be set.
256+
* If user records any tracing event the `Span` object needs to know if the data are kept or not.
257+
It may be possible to always collect all the events until the sampling decision is made but this is
258+
an important optimization.
259+
260+
There are two important use-cases to be considered:
261+
* All information that may be used for sampling decisions are available at the moment when the
262+
logical `Span` operation should start. This is the most common case.
263+
* Some information that may be used for sampling decision are NOT available at the moment when the
264+
logical `Span` operation should start (e.g. `http.route` may be determine later).
265+
266+
The current [span creation logic][span-creation] facilitates very well the first use-case, but
267+
the second use-case requires users to record the logical `start_time` and collect all the
268+
information necessarily to start the `Span` in custom objects, then when all the properties are
269+
available call the span creation API.
270+
271+
The RFC proposes that we keep the current [span creation logic][span-creation] as it is and we will
272+
address the delayed sampling in a different RFC when that becomes a high priority.
273+
274+
The SDK must call the `Sampler` every time a `Span` is created during the start span operation.
275+
276+
**Alternatives considerations:**
277+
* We considered, to offer a delayed span construction mechanism:
278+
* For languages where a `Builder` pattern is used to construct a `Span`, to allow users to
279+
create a `Builder` where the start time of the Span is considered when the `Builder` is created.
280+
* For languages where no intermediate object is used to construct a `Span`, to allow users maybe
281+
via a `StartSpanOption` object to start a `Span`. The `StartSpanOption` allows users to set all
282+
the start `Span` properties.
283+
* Pros:
284+
* Would resolve the second use-case posted above.
285+
* Cons:
286+
* We could not identify too many real case examples for the second use-case and decided to
287+
postpone the decision to avoid premature decisions.
288+
* We considered, instead of requiring that sampling decision happens before the `Span` is
289+
created to add an explicit `MakeSamplingDecision(SamplingHint)` on the `Span`. Attempts to create
290+
a child `Span`, or to access the `SpanContext` would fail if `MakeSamplingDecision()` had not yet
291+
been run.
292+
* Pros:
293+
* Simplifies the case when all the attributes that may be used for sampling are not available
294+
when the logical `Span` operation should start.
295+
* Cons:
296+
* The most common case would have required an extra API call.
297+
* Error prone, users may forget to call the extra API.
298+
* Unexpected and hard to find errors if user tries to create a child `Span` before calling
299+
MakeSamplingDecision().
300+
* We considered allowing the sampling decision to be arbitrarily delayed, but guaranteed before
301+
any child `Span` is created, or `SpanContext` is accessed, or before `Span.end()` finished.
302+
* Pros:
303+
* Similar and smaller API that supports both use-cases defined ahead.
304+
* Cons:
305+
* If `SamplingHint` needs to also be delayed recorded then an extra API on Span is required
306+
to set this.
307+
* Does not allow optimization to not record tracing events, all tracing events MUST be
308+
recorded before the sampling decision is made.
309+
310+
## Prior art and alternatives
311+
Prior art for Zipkin, and other Dapper based systems: all client-side sampling decisions are made at
312+
head. Thus, we need to retain compatibility with this.
313+
314+
## Open questions
315+
This RFC does not necessarily resolve the question of how to propagate sampling rate values between
316+
different spans and processes. A separate RFC will be opened to cover this case.
317+
318+
## Future possibilities
319+
In the future, we propose that library developers may be able to defer the decision on whether to
320+
recommend the trace be sampled or not sampled until mid-way through execution;
321+
322+
## Related Issues
323+
* [opentelemetry-specification/189](https://github.com/open-telemetry/opentelemetry-specification/issues/189)
324+
* [opentelemetry-specification/187](https://github.com/open-telemetry/opentelemetry-specification/issues/187)
325+
* [opentelemetry-specification/164](https://github.com/open-telemetry/opentelemetry-specification/issues/164)
326+
* [opentelemetry-specification/125](https://github.com/open-telemetry/opentelemetry-specification/issues/125)
327+
* [opentelemetry-specification/87](https://github.com/open-telemetry/opentelemetry-specification/issues/87)
328+
* [opentelemetry-specification/66](https://github.com/open-telemetry/opentelemetry-specification/issues/66)
329+
* [opentelemetry-specification/65](https://github.com/open-telemetry/opentelemetry-specification/issues/65)
330+
* [opentelemetry-specification/53](https://github.com/open-telemetry/opentelemetry-specification/issues/53)
331+
* [opentelemetry-specification/33](https://github.com/open-telemetry/opentelemetry-specification/issues/33)
332+
* [opentelemetry-specification/32](https://github.com/open-telemetry/opentelemetry-specification/issues/32)
333+
* [opentelemetry-specification/31](https://github.com/open-telemetry/opentelemetry-specification/issues/31)
334+
335+
[trace-flags]: https://github.com/w3c/trace-context/blob/master/spec/20-http_header_format.md#trace-flags
336+
[specs-pr-216]: https://github.com/open-telemetry/opentelemetry-specification/pull/216
337+
[span-creation]: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-tracing.md#span-creation

0 commit comments

Comments
 (0)