Rework Kotlin coroutines instrumentation around coroutine context #8774

mcculls · 2025-05-06T21:17:58Z

What Does This Do

The previous approach associated the scope state with the job (coroutine) instance using field-injection. However this doesn't work when the coroutine instance might be recreated or wrapped by something else.

This new approach creates a DatadogThreadContextElement for every coroutine context created. The scope state is then lazily captured within this element: either after non-lazy coroutines are constructed, when lazy coroutines start, or failing that when the element is first used to update the thread context.

At the same time it captures the active span using a continuation to avoid the trace from completing before the coroutine is complete. Note there is no need to (re)activate the continuation when switching contexts because we have already captured the scope state.

Motivation

This simpler instrumentation will be easier to move over to the new Context API

This also fixes the scenario described in https://github.com/arun-rama/KotlinCoroutineTracingIssueRepro/tree/main

Additional Notes

As an added benefit, this approach does not require field injection

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: AIDM-602

pr-commenter · 2025-05-06T22:09:04Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/rework-kotlin-instrumentation
git_commit_date	1746825821	1746995032
git_commit_sha	`f494c33`	`91445ae`
release_version	1.50.0-SNAPSHOT~f494c33b01	1.50.0-SNAPSHOT~91445aea2f

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746997446	1746997446
ci_job_id	932726370	932726370
ci_pipeline_id	64734156	64734156
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-i83qnnyx-project-304-concurrent-1-9h633128 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-i83qnnyx-project-304-concurrent-1-9h633128 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.023 s) : 0, 1023211
Total [baseline] (10.479 s) : 0, 10479305
Agent [candidate] (1.018 s) : 0, 1018140
Total [candidate] (10.494 s) : 0, 10493633
section appsec
Agent [baseline] (1.157 s) : 0, 1157173
Total [baseline] (10.653 s) : 0, 10653479
Agent [candidate] (1.161 s) : 0, 1160889
Total [candidate] (10.634 s) : 0, 10634415
section iast
Agent [baseline] (1.147 s) : 0, 1146691
Total [baseline] (10.867 s) : 0, 10867323
Agent [candidate] (1.146 s) : 0, 1146259
Total [candidate] (10.93 s) : 0, 10929681
section profiling
Agent [baseline] (1.281 s) : 0, 1280534
Total [baseline] (10.898 s) : 0, 10898256
Agent [candidate] (1.28 s) : 0, 1279562
Total [candidate] (10.807 s) : 0, 10807447

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.023 s	-
Agent	appsec	1.157 s	133.962 ms (13.1%)
Agent	iast	1.147 s	123.48 ms (12.1%)
Agent	profiling	1.281 s	257.323 ms (25.1%)
Total	tracing	10.479 s	-
Total	appsec	10.653 s	174.174 ms (1.7%)
Total	iast	10.867 s	388.019 ms (3.7%)
Total	profiling	10.898 s	418.951 ms (4.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.018 s	-
Agent	appsec	1.161 s	142.749 ms (14.0%)
Agent	iast	1.146 s	128.119 ms (12.6%)
Agent	profiling	1.28 s	261.422 ms (25.7%)
Total	tracing	10.494 s	-
Total	appsec	10.634 s	140.782 ms (1.3%)
Total	iast	10.93 s	436.048 ms (4.2%)
Total	profiling	10.807 s	313.813 ms (3.0%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (685.953 ms) : 0, 685953
BytebuddyAgent [candidate] (681.773 ms) : 0, 681773
GlobalTracer [baseline] (240.971 ms) : 0, 240971
GlobalTracer [candidate] (239.794 ms) : 0, 239794
AppSec [baseline] (54.627 ms) : 0, 54627
AppSec [candidate] (54.479 ms) : 0, 54479
Debugger [baseline] (8.962 ms) : 0, 8962
Debugger [candidate] (8.192 ms) : 0, 8192
Remote Config [baseline] (691.447 µs) : 0, 691
Remote Config [candidate] (694.489 µs) : 0, 694
Telemetry [baseline] (8.343 ms) : 0, 8343
Telemetry [candidate] (9.696 ms) : 0, 9696
section appsec
BytebuddyAgent [baseline] (698.813 ms) : 0, 698813
BytebuddyAgent [candidate] (700.179 ms) : 0, 700179
GlobalTracer [baseline] (235.887 ms) : 0, 235887
GlobalTracer [candidate] (236.585 ms) : 0, 236585
AppSec [baseline] (174.151 ms) : 0, 174151
AppSec [candidate] (175.858 ms) : 0, 175858
Debugger [baseline] (5.943 ms) : 0, 5943
Debugger [candidate] (5.957 ms) : 0, 5957
Remote Config [baseline] (618.861 µs) : 0, 619
Remote Config [candidate] (631.117 µs) : 0, 631
Telemetry [baseline] (7.411 ms) : 0, 7411
Telemetry [candidate] (7.418 ms) : 0, 7418
IAST [baseline] (21.673 ms) : 0, 21673
IAST [candidate] (21.635 ms) : 0, 21635
section iast
BytebuddyAgent [baseline] (800.43 ms) : 0, 800430
BytebuddyAgent [candidate] (800.101 ms) : 0, 800101
GlobalTracer [baseline] (230.077 ms) : 0, 230077
GlobalTracer [candidate] (230.244 ms) : 0, 230244
AppSec [baseline] (52.0 ms) : 0, 52000
AppSec [candidate] (48.499 ms) : 0, 48499
Debugger [baseline] (5.902 ms) : 0, 5902
Debugger [candidate] (5.859 ms) : 0, 5859
Remote Config [baseline] (594.637 µs) : 0, 595
Remote Config [candidate] (591.026 µs) : 0, 591
Telemetry [baseline] (7.87 ms) : 0, 7870
Telemetry [candidate] (7.879 ms) : 0, 7879
IAST [baseline] (26.435 ms) : 0, 26435
IAST [candidate] (29.686 ms) : 0, 29686
section profiling
BytebuddyAgent [baseline] (673.276 ms) : 0, 673276
BytebuddyAgent [candidate] (672.336 ms) : 0, 672336
GlobalTracer [baseline] (378.533 ms) : 0, 378533
GlobalTracer [candidate] (378.532 ms) : 0, 378532
AppSec [baseline] (54.242 ms) : 0, 54242
AppSec [candidate] (53.949 ms) : 0, 53949
Debugger [baseline] (6.085 ms) : 0, 6085
Debugger [candidate] (6.165 ms) : 0, 6165
Remote Config [baseline] (645.496 µs) : 0, 645
Remote Config [candidate] (651.493 µs) : 0, 651
Telemetry [baseline] (8.089 ms) : 0, 8089
Telemetry [candidate] (8.121 ms) : 0, 8121
ProfilingAgent [baseline] (109.069 ms) : 0, 109069
ProfilingAgent [candidate] (109.283 ms) : 0, 109283
Profiling [baseline] (109.094 ms) : 0, 109094
Profiling [candidate] (109.309 ms) : 0, 109309

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.029 s) : 0, 1028787
Total [baseline] (8.633 s) : 0, 8633479
Agent [candidate] (1.019 s) : 0, 1018982
Total [candidate] (8.645 s) : 0, 8645096
section iast
Agent [baseline] (1.146 s) : 0, 1145648
Total [baseline] (9.188 s) : 0, 9188155
Agent [candidate] (1.147 s) : 0, 1146884
Total [candidate] (9.2 s) : 0, 9199738
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.147 s) : 0, 1147246
Total [baseline] (9.227 s) : 0, 9227232
Agent [candidate] (1.153 s) : 0, 1153361
Total [candidate] (9.205 s) : 0, 9205317
section iast_TELEMETRY_OFF
Agent [baseline] (1.161 s) : 0, 1161068
Total [baseline] (9.266 s) : 0, 9266352
Agent [candidate] (1.144 s) : 0, 1143715
Total [candidate] (9.196 s) : 0, 9196216

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.029 s	-
Agent	iast	1.146 s	116.861 ms (11.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.147 s	118.459 ms (11.5%)
Agent	iast_TELEMETRY_OFF	1.161 s	132.281 ms (12.9%)
Total	tracing	8.633 s	-
Total	iast	9.188 s	554.676 ms (6.4%)
Total	iast_HARDCODED_SECRET_DISABLED	9.227 s	593.753 ms (6.9%)
Total	iast_TELEMETRY_OFF	9.266 s	632.872 ms (7.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.019 s	-
Agent	iast	1.147 s	127.901 ms (12.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.153 s	134.379 ms (13.2%)
Agent	iast_TELEMETRY_OFF	1.144 s	124.733 ms (12.2%)
Total	tracing	8.645 s	-
Total	iast	9.2 s	554.642 ms (6.4%)
Total	iast_HARDCODED_SECRET_DISABLED	9.205 s	560.221 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.196 s	551.119 ms (6.4%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.943 ms) : 0, 688943
BytebuddyAgent [candidate] (680.45 ms) : 0, 680450
GlobalTracer [baseline] (241.699 ms) : 0, 241699
GlobalTracer [candidate] (239.245 ms) : 0, 239245
AppSec [baseline] (55.7 ms) : 0, 55700
AppSec [candidate] (54.81 ms) : 0, 54810
Debugger [baseline] (8.311 ms) : 0, 8311
Debugger [candidate] (10.569 ms) : 0, 10569
Remote Config [baseline] (694.674 µs) : 0, 695
Remote Config [candidate] (686.604 µs) : 0, 687
Telemetry [baseline] (9.774 ms) : 0, 9774
Telemetry [candidate] (9.684 ms) : 0, 9684
section iast
BytebuddyAgent [baseline] (799.644 ms) : 0, 799644
BytebuddyAgent [candidate] (800.234 ms) : 0, 800234
GlobalTracer [baseline] (229.688 ms) : 0, 229688
GlobalTracer [candidate] (230.279 ms) : 0, 230279
AppSec [baseline] (50.279 ms) : 0, 50279
AppSec [candidate] (51.079 ms) : 0, 51079
Debugger [baseline] (5.907 ms) : 0, 5907
Debugger [candidate] (5.906 ms) : 0, 5906
Remote Config [baseline] (584.316 µs) : 0, 584
Remote Config [candidate] (606.098 µs) : 0, 606
Telemetry [baseline] (7.845 ms) : 0, 7845
Telemetry [candidate] (7.906 ms) : 0, 7906
IAST [baseline] (28.171 ms) : 0, 28171
IAST [candidate] (24.322 ms) : 0, 24322
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (801.014 ms) : 0, 801014
BytebuddyAgent [candidate] (804.888 ms) : 0, 804888
GlobalTracer [baseline] (230.219 ms) : 0, 230219
GlobalTracer [candidate] (231.627 ms) : 0, 231627
AppSec [baseline] (48.567 ms) : 0, 48567
AppSec [candidate] (48.688 ms) : 0, 48688
Debugger [baseline] (5.853 ms) : 0, 5853
Debugger [candidate] (5.941 ms) : 0, 5941
Remote Config [baseline] (579.168 µs) : 0, 579
Remote Config [candidate] (586.448 µs) : 0, 586
Telemetry [baseline] (7.843 ms) : 0, 7843
Telemetry [candidate] (8.011 ms) : 0, 8011
IAST [baseline] (29.696 ms) : 0, 29696
IAST [candidate] (30.084 ms) : 0, 30084
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (810.262 ms) : 0, 810262
BytebuddyAgent [candidate] (796.787 ms) : 0, 796787
GlobalTracer [baseline] (232.98 ms) : 0, 232980
GlobalTracer [candidate] (230.833 ms) : 0, 230833
AppSec [baseline] (54.19 ms) : 0, 54190
AppSec [candidate] (53.319 ms) : 0, 53319
Debugger [baseline] (6.083 ms) : 0, 6083
Debugger [candidate] (5.991 ms) : 0, 5991
Remote Config [baseline] (617.799 µs) : 0, 618
Remote Config [candidate] (608.084 µs) : 0, 608
Telemetry [baseline] (7.952 ms) : 0, 7952
Telemetry [candidate] (7.858 ms) : 0, 7858
IAST [baseline] (25.156 ms) : 0, 25156
IAST [candidate] (24.847 ms) : 0, 24847

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-11T20:34:28	2025-05-11T20:42:10
git_branch	master	mcculls/rework-kotlin-instrumentation
git_commit_date	1746825821	1746995032
git_commit_sha	`f494c33`	`91445ae`
release_version	1.50.0-SNAPSHOT~f494c33b01	1.50.0-SNAPSHOT~91445aea2f
start_time	2025-05-11T20:34:14	2025-05-11T20:41:56

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746996527	1746996527
ci_job_id	932726371	932726371
ci_pipeline_id	64734156	64734156
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-psrqmy6d-project-304-concurrent-0-kpb4347j 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-psrqmy6d-project-304-concurrent-0-kpb4347j 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01
    dateFormat X
    axisFormat %s
section baseline
no_agent (381.245 µs) : 362, 401
.   : milestone, 381,
iast (530.203 µs) : 507, 553
.   : milestone, 530,
iast_FULL (724.13 µs) : 702, 746
.   : milestone, 724,
iast_GLOBAL (553.688 µs) : 532, 575
.   : milestone, 554,
iast_HARDCODED_SECRET_DISABLED (516.875 µs) : 494, 539
.   : milestone, 517,
iast_INACTIVE (455.373 µs) : 434, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (510.831 µs) : 487, 534
.   : milestone, 511,
tracing (463.084 µs) : 441, 485
.   : milestone, 463,
section candidate
no_agent (385.554 µs) : 366, 405
.   : milestone, 386,
iast (513.126 µs) : 491, 535
.   : milestone, 513,
iast_FULL (740.995 µs) : 719, 763
.   : milestone, 741,
iast_GLOBAL (562.872 µs) : 540, 585
.   : milestone, 563,
iast_HARDCODED_SECRET_DISABLED (522.818 µs) : 500, 546
.   : milestone, 523,
iast_INACTIVE (465.449 µs) : 443, 488
.   : milestone, 465,
iast_TELEMETRY_OFF (510.316 µs) : 487, 533
.   : milestone, 510,
tracing (457.472 µs) : 435, 480
.   : milestone, 457,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	381.245 µs [361.677 µs, 400.813 µs]	-
iast	530.203 µs [506.996 µs, 553.41 µs]	148.958 µs (39.1%)
iast_FULL	724.13 µs [702.19 µs, 746.071 µs]	342.885 µs (89.9%)
iast_GLOBAL	553.688 µs [532.17 µs, 575.207 µs]	172.443 µs (45.2%)
iast_HARDCODED_SECRET_DISABLED	516.875 µs [494.441 µs, 539.309 µs]	135.63 µs (35.6%)
iast_INACTIVE	455.373 µs [434.403 µs, 476.344 µs]	74.128 µs (19.4%)
iast_TELEMETRY_OFF	510.831 µs [487.475 µs, 534.186 µs]	129.586 µs (34.0%)
tracing	463.084 µs [440.839 µs, 485.329 µs]	81.839 µs (21.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	385.554 µs [365.728 µs, 405.38 µs]	-
iast	513.126 µs [491.38 µs, 534.872 µs]	127.572 µs (33.1%)
iast_FULL	740.995 µs [718.958 µs, 763.031 µs]	355.441 µs (92.2%)
iast_GLOBAL	562.872 µs [540.31 µs, 585.435 µs]	177.318 µs (46.0%)
iast_HARDCODED_SECRET_DISABLED	522.818 µs [499.641 µs, 545.995 µs]	137.264 µs (35.6%)
iast_INACTIVE	465.449 µs [442.796 µs, 488.102 µs]	79.895 µs (20.7%)
iast_TELEMETRY_OFF	510.316 µs [487.17 µs, 533.462 µs]	124.762 µs (32.4%)
tracing	457.472 µs [435.386 µs, 479.558 µs]	71.918 µs (18.7%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.359 ms) : 1340, 1378
.   : milestone, 1359,
appsec (1.742 ms) : 1718, 1765
.   : milestone, 1742,
appsec_no_iast (1.732 ms) : 1709, 1754
.   : milestone, 1732,
code_origins (1.643 ms) : 1617, 1669
.   : milestone, 1643,
iast (1.509 ms) : 1485, 1533
.   : milestone, 1509,
profiling (1.521 ms) : 1496, 1546
.   : milestone, 1521,
tracing (1.484 ms) : 1459, 1509
.   : milestone, 1484,
section candidate
no_agent (1.366 ms) : 1347, 1385
.   : milestone, 1366,
appsec (1.749 ms) : 1726, 1773
.   : milestone, 1749,
appsec_no_iast (1.708 ms) : 1684, 1732
.   : milestone, 1708,
code_origins (1.68 ms) : 1652, 1708
.   : milestone, 1680,
iast (1.52 ms) : 1497, 1544
.   : milestone, 1520,
profiling (1.527 ms) : 1503, 1550
.   : milestone, 1527,
tracing (1.507 ms) : 1482, 1532
.   : milestone, 1507,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.359 ms [1.34 ms, 1.378 ms]	-
appsec	1.742 ms [1.718 ms, 1.765 ms]	382.301 µs (28.1%)
appsec_no_iast	1.732 ms [1.709 ms, 1.754 ms]	372.345 µs (27.4%)
code_origins	1.643 ms [1.617 ms, 1.669 ms]	283.951 µs (20.9%)
iast	1.509 ms [1.485 ms, 1.533 ms]	149.884 µs (11.0%)
profiling	1.521 ms [1.496 ms, 1.546 ms]	161.605 µs (11.9%)
tracing	1.484 ms [1.459 ms, 1.509 ms]	124.7 µs (9.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.366 ms [1.347 ms, 1.385 ms]	-
appsec	1.749 ms [1.726 ms, 1.773 ms]	383.611 µs (28.1%)
appsec_no_iast	1.708 ms [1.684 ms, 1.732 ms]	342.24 µs (25.1%)
code_origins	1.68 ms [1.652 ms, 1.708 ms]	314.468 µs (23.0%)
iast	1.52 ms [1.497 ms, 1.544 ms]	154.51 µs (11.3%)
profiling	1.527 ms [1.503 ms, 1.55 ms]	161.088 µs (11.8%)
tracing	1.507 ms [1.482 ms, 1.532 ms]	141.465 µs (10.4%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/rework-kotlin-instrumentation
git_commit_date	1746825821	1746995032
git_commit_sha	`f494c33`	`91445ae`
release_version	1.50.0-SNAPSHOT~f494c33b01	1.50.0-SNAPSHOT~91445aea2f

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1746996972	1746996972
ci_job_id	932726372	932726372
ci_pipeline_id	64734156	64734156
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-i83qnnyx-project-304-concurrent-2-1mg3z1um 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-i83qnnyx-project-304-concurrent-2-1mg3z1um 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.417 s) : 15417000, 15417000
.   : milestone, 15417000,
appsec (15.092 s) : 15092000, 15092000
.   : milestone, 15092000,
iast (19.075 s) : 19075000, 19075000
.   : milestone, 19075000,
iast_GLOBAL (18.188 s) : 18188000, 18188000
.   : milestone, 18188000,
profiling (15.652 s) : 15652000, 15652000
.   : milestone, 15652000,
tracing (14.921 s) : 14921000, 14921000
.   : milestone, 14921000,
section candidate
no_agent (14.962 s) : 14962000, 14962000
.   : milestone, 14962000,
appsec (15.146 s) : 15146000, 15146000
.   : milestone, 15146000,
iast (18.987 s) : 18987000, 18987000
.   : milestone, 18987000,
iast_GLOBAL (18.003 s) : 18003000, 18003000
.   : milestone, 18003000,
profiling (15.502 s) : 15502000, 15502000
.   : milestone, 15502000,
tracing (15.045 s) : 15045000, 15045000
.   : milestone, 15045000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.417 s [15.417 s, 15.417 s]	-
appsec	15.092 s [15.092 s, 15.092 s]	-325.0 ms (-2.1%)
iast	19.075 s [19.075 s, 19.075 s]	3.658 s (23.7%)
iast_GLOBAL	18.188 s [18.188 s, 18.188 s]	2.771 s (18.0%)
profiling	15.652 s [15.652 s, 15.652 s]	235.0 ms (1.5%)
tracing	14.921 s [14.921 s, 14.921 s]	-496.0 ms (-3.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.962 s [14.962 s, 14.962 s]	-
appsec	15.146 s [15.146 s, 15.146 s]	184.0 ms (1.2%)
iast	18.987 s [18.987 s, 18.987 s]	4.025 s (26.9%)
iast_GLOBAL	18.003 s [18.003 s, 18.003 s]	3.041 s (20.3%)
profiling	15.502 s [15.502 s, 15.502 s]	540.0 ms (3.6%)
tracing	15.045 s [15.045 s, 15.045 s]	83.0 ms (0.6%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~91445aea2f, baseline=1.50.0-SNAPSHOT~f494c33b01
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.483 ms) : 1471, 1495
.   : milestone, 1483,
appsec (2.418 ms) : 2368, 2467
.   : milestone, 2418,
iast (2.202 ms) : 2140, 2264
.   : milestone, 2202,
iast_GLOBAL (2.233 ms) : 2171, 2295
.   : milestone, 2233,
profiling (2.05 ms) : 2000, 2100
.   : milestone, 2050,
tracing (2.033 ms) : 1984, 2081
.   : milestone, 2033,
section candidate
no_agent (1.478 ms) : 1467, 1490
.   : milestone, 1478,
appsec (2.418 ms) : 2369, 2468
.   : milestone, 2418,
iast (2.204 ms) : 2142, 2266
.   : milestone, 2204,
iast_GLOBAL (2.248 ms) : 2185, 2311
.   : milestone, 2248,
profiling (2.041 ms) : 1992, 2090
.   : milestone, 2041,
tracing (2.028 ms) : 1980, 2076
.   : milestone, 2028,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.483 ms [1.471 ms, 1.495 ms]	-
appsec	2.418 ms [2.368 ms, 2.467 ms]	934.494 µs (63.0%)
iast	2.202 ms [2.14 ms, 2.264 ms]	718.712 µs (48.5%)
iast_GLOBAL	2.233 ms [2.171 ms, 2.295 ms]	749.783 µs (50.6%)
profiling	2.05 ms [2.0 ms, 2.1 ms]	567.034 µs (38.2%)
tracing	2.033 ms [1.984 ms, 2.081 ms]	549.694 µs (37.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.478 ms [1.467 ms, 1.49 ms]	-
appsec	2.418 ms [2.369 ms, 2.468 ms]	940.276 µs (63.6%)
iast	2.204 ms [2.142 ms, 2.266 ms]	725.574 µs (49.1%)
iast_GLOBAL	2.248 ms [2.185 ms, 2.311 ms]	770.175 µs (52.1%)
profiling	2.041 ms [1.992 ms, 2.09 ms]	562.725 µs (38.1%)
tracing	2.028 ms [1.98 ms, 2.076 ms]	549.922 µs (37.2%)

As an added benefit, this approach does not require field injection

amarziali

This is a great contribution. Much cleaner than before. Thanks also for the detailed description

PerfectSlayer

Nice improvement! 👍

...og/trace/instrumentation/kotlin/coroutines/AbstractKotlinCoroutineInstrumentationTest.groovy

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.49.0` -> `1.50.0` | --- ### Release Notes <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.50.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.50.0): 1.50.0 ### Deprecation Notice > \[!NOTE] > `DD_RUNTIME_ID_ENABLED` has been deprecated and will be removed in future releases. Please use `DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED` instead. ### Components #### Application Security Management (WAF) - 🐛 Add String length truncation limit to ObjectIntrospector and update truncation metrics ([#8825](DataDog/dd-trace-java#8825) - [@jandro996](https://github.com/jandro996)) - 🐛 Adapt standalone ASM to support API Security ([#8804](DataDog/dd-trace-java#8804) - [@jandro996](https://github.com/jandro996)) - ✨ Add appsec.waf.input\_truncated metric ([#8791](DataDog/dd-trace-java#8791) - [@jandro996](https://github.com/jandro996)) - ✨ Extended appsec request body collection ([#8748](DataDog/dd-trace-java#8748) - [@jandro996](https://github.com/jandro996)) - ✨ Extended appsec request/response headers collection ([#8724](DataDog/dd-trace-java#8724) - [@jandro996](https://github.com/jandro996)) #### Build & Tooling - ✨ Add artifacts to public s3 bucket ([#8947](DataDog/dd-trace-java#8947) - [@randomanderson](https://github.com/randomanderson)) #### Continuous Integration Visibility - ✨ Improve PR information building ([#8908](DataDog/dd-trace-java#8908) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Truncate span stack traces when Test Optimization is enabled ([#8903](DataDog/dd-trace-java#8903) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Ensure auto-detected service name is the same for every process in the same build ([#8902](DataDog/dd-trace-java#8902) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Use tag as fallback in api requests if no branch is available ([#8876](DataDog/dd-trace-java#8876) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Add support for JUnit 5.13-RC1 ([#8865](DataDog/dd-trace-java#8865), [#8871](DataDog/dd-trace-java#8871) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement attempt to fix v3 and v4 and bump capability version ([#8824](DataDog/dd-trace-java#8824) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - 🧹 Align retry logic for all test framework instrumentations ([#8803](DataDog/dd-trace-java#8803) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - 🐛 Always build ci workspace without trailing separator ([#8788](DataDog/dd-trace-java#8788) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Add commit discrepancies telemetry when building repository git information ([#8763](DataDog/dd-trace-java#8763) - [@daniel-mohedano](https://github.com/daniel-mohedano)) #### Data Streams Monitoring - 💡 Surface process tags in dsm payloads and use them for base hash calculation ([#8836](DataDog/dd-trace-java#8836) - [@amarziali](https://github.com/amarziali)) #### Dynamic Instrumentation - ✨ Optimized allocations for collection filter functions ([#8896](DataDog/dd-trace-java#8896) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix SymDB upload size check ([#8887](DataDog/dd-trace-java#8887) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Add support for Set in filter function ([#8873](DataDog/dd-trace-java#8873) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Add support for isDefined in log template ([#8859](DataDog/dd-trace-java#8859) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix Max captured frames for Exception Replay ([#8856](DataDog/dd-trace-java#8856) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Remove static inherited fields collection ([#8832](DataDog/dd-trace-java#8832) - [@jpbempel](https://github.com/jpbempel)) - 💡 Add process tags to dynamic instrumentation intake payload ([#8779](DataDog/dd-trace-java#8779) - [@amarziali](https://github.com/amarziali)) #### GraalVM native-image - ✨ Add support for GraalVM Native GC metrics ([#8913](DataDog/dd-trace-java#8913) - [@ygree](https://github.com/ygree)) - ✨ Add JMXFetch support for GraalVM Native ([#8569](DataDog/dd-trace-java#8569) - [@ygree](https://github.com/ygree)) #### JMX fetch - ✨ Add support for GraalVM Native GC metrics ([#8913](DataDog/dd-trace-java#8913) - [@ygree](https://github.com/ygree)) #### Library Injection - ✨ Deny oracle db jvm based tools ([#8909](DataDog/dd-trace-java#8909) - [@bric3](https://github.com/bric3)) #### OpenTracing - 🐛 Fix OT packaging for exception replay ([#8912](DataDog/dd-trace-java#8912) - [@jpbempel](https://github.com/jpbempel)) #### Profiling - ✨ Bump ddprof to 1.27.0 ([#8893](DataDog/dd-trace-java#8893) - [@jbachorik](https://github.com/jbachorik)) - Properly handle the adaptive sampling interval overflow by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#213 - Fix [#200](DataDog/dd-trace-java#200) Crash related to aligned\_alloc and free in context by [@yanglong1010](https://github.com/yanglong1010) in DataDog/java-profiler#208 - Explicitly initialize empty context page by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#210 - Re-connect crash recursion protection with VM stackwalker by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#214 - ✨ Enable ZSTD compression for profiling ([#8862](DataDog/dd-trace-java#8862) - [@MattAlp](https://github.com/MattAlp)) - ✨ Extend JPS re-implementation to J9 family ([#8813](DataDog/dd-trace-java#8813) - [@MattAlp](https://github.com/MattAlp)) - 💡 Collect process tags for profiling upload requests ([#8780](DataDog/dd-trace-java#8780) - [@amarziali](https://github.com/amarziali)) #### Telemetry - 💡 Surface process tags on telemetry payloads ([#8837](DataDog/dd-trace-java#8837) - [@amarziali](https://github.com/amarziali)) #### Trace context propagation - ✨ Migrating all HttpClient Instrumentations to Inject Full Context ([#8826](DataDog/dd-trace-java#8826) - [@mhlidd](https://github.com/mhlidd)) - ✨ Migrating all HttpServer Instrumentations to Extract full Context ([#8820](DataDog/dd-trace-java#8820) - [@mhlidd](https://github.com/mhlidd)) - ✨ Add context API support OTel propagators ([#8770](DataDog/dd-trace-java#8770) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Tracer core - ✨⚡ Skip JAXB generated classes classloader ([#9003](DataDog/dd-trace-java#9003) - [@bric3](https://github.com/bric3)) - ✨ Add DD\_RUNTIME\_METRICS\_RUNTIME\_ID\_ENABLED alias for runtime id generation ([#8981](DataDog/dd-trace-java#8981) - [@amarziali](https://github.com/amarziali)) - 🐛 Use resolved address for peer.hostname when available without hitting the cache ([#8915](DataDog/dd-trace-java#8915) - [@amarziali](https://github.com/amarziali)) - 💡 Surface server name process tag for tomcat ([#8894](DataDog/dd-trace-java#8894) - [@amarziali](https://github.com/amarziali)) - 💡 Surface websphere cell and server name on process tags ([#8880](DataDog/dd-trace-java#8880) - [@amarziali](https://github.com/amarziali)) - ✨ Added special lightweight pre-main class that skips installation on incompatible JVMs. ([#8855](DataDog/dd-trace-java#8855) - [@AlexeyKuznetsov-DD](https://github.com/AlexeyKuznetsov-DD)) - 💡 Add entrypoint type to process tags ([#8839](DataDog/dd-trace-java#8839) - [@amarziali](https://github.com/amarziali)) - ✨ Extend JPS re-implementation to J9 family ([#8813](DataDog/dd-trace-java#8813) - [@MattAlp](https://github.com/MattAlp)) - ✨ Notify listeners when the scope top changes after switching scope stacks ([#8797](DataDog/dd-trace-java#8797) - [@mcculls](https://github.com/mcculls)) - ✨ Read hsperfdata for Java PIDs if jvmstat is unavailable ([#8792](DataDog/dd-trace-java#8792) - [@MattAlp](https://github.com/MattAlp)) - 🐛 Turn JDK socket support on by default ([#8752](DataDog/dd-trace-java#8752) - [@sarahchen6](https://github.com/sarahchen6)) - ✨ Simplify context propagation ([#8719](DataDog/dd-trace-java#8719) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Add JSON parsing support ([#8579](DataDog/dd-trace-java#8579) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Tracer internal logging - ✨ Fix printing format of span identifiers ([#8897](DataDog/dd-trace-java#8897) - [@vandonr](https://github.com/vandonr)) #### Tracer public API - 💡 Track the source of installation ([#8956](DataDog/dd-trace-java#8956) - [@mabdinur](https://github.com/mabdinur)) - ✨ Enforce size limit on application\_monitoring.yaml files ([#8789](DataDog/dd-trace-java#8789) - [@mtoffl01](https://github.com/mtoffl01)) - ✨ Enabling baggage cache to support limits and non-ascii characters ([#8713](DataDog/dd-trace-java#8713) - [@mhlidd](https://github.com/mhlidd)) ### Instrumentations #### AWS Lambda instrumentation - ✨ Pass Lambda Request ID to Extension ([#8814](DataDog/dd-trace-java#8814) - [@nhulston](https://github.com/nhulston)) #### Core Java language instrumentation - ✨ Ensure ClassloadingInstrumentation is always applied even with `DD_TRACE_ENABLED=false` ([#8863](DataDog/dd-trace-java#8863) - [@mcculls](https://github.com/mcculls)) #### Eclipse Vert.x instrumentation - 🐛 Do not override route with / in vertx instrumentation ([#8881](DataDog/dd-trace-java#8881) - [@vandonr](https://github.com/vandonr)) #### IBM Liberty - 🐛 Fix error mark on http status for IBM liberty ([#8822](DataDog/dd-trace-java#8822) - [@amarziali](https://github.com/amarziali)) #### JDBC instrumentation - 🐛 Do not prepend DBM <> APM trace comment in SQLCommenter if there is a pg plan hint ([#8864](DataDog/dd-trace-java#8864) - [@edengorevoy](https://github.com/edengorevoy)) #### JMS instrumentation - ✨ Add jms as an extra integration name where there is JMS involved ([#8933](DataDog/dd-trace-java#8933) - [@vandonr](https://github.com/vandonr)) #### Kotlin instrumentation - ✨ Enable kotlin\_coroutine integration by default ([#8848](DataDog/dd-trace-java#8848) - [@mcculls](https://github.com/mcculls)) - 🧹 Rework Kotlin coroutines instrumentation around coroutine context ([#8774](DataDog/dd-trace-java#8774) - [@mcculls](https://github.com/mcculls)) #### OpenTelemetry instrumentation - 🐛 Support WithSpan inheritContext attribute ([#8858](DataDog/dd-trace-java#8858) - [@amarziali](https://github.com/amarziali)) - ✨ Add context API support OTel propagators ([#8770](DataDog/dd-trace-java#8770) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Play Framework instrumentation - 🐛 Fix the Play Framework's span resource name priority so that the client JAX-RS 404 cannot override it ([#8591](DataDog/dd-trace-java#8591) - [@ygree](https://github.com/ygree)) #### Quarkus Instrumentation - 🐛 Ignore quarkus jaxrs stubs and cdi wrapper proxies ([#8891](DataDog/dd-trace-java#8891) - [@amarziali](https://github.com/amarziali)) #### ServiceTalk - ✨ Improve ServiceTalk Captured Context API Instrumentation for v0.42.56+ ([#8821](DataDog/dd-trace-java#8821) - [@ygree](https://github.com/ygree)) #### Spring instrumentation - ✨ Supporting Baggage for Instrumentations used in Weblog Tests ([#8773](DataDog/dd-trace-java#8773) - [@mhlidd](https://github.com/mhlidd)) #### WebSocket Instrumentation - 💡 Trace websocket for spring webflux reactive handlers ([#8831](DataDog/dd-trace-java#8831) - [@amarziali](https://github.com/amarziali)) - 💡:test\_tube: WebSocket support for Netty ([#8632](DataDog/dd-trace-java#8632) - [@ValentinZakharov](https://github.com/ValentinZakharov)) #### Zio Instrumentation - 🧹 Cleanup Zio fiber instrumentation to avoid repeated activation of continuation ([#8798](DataDog/dd-trace-java#8798) - [@mcculls](https://github.com/mcculls)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 9207366cdb6a1bd098082305d354a0a3c4622d7a

mcculls added tag: do not merge Do not merge changes type: refactoring inst: kotlin Kotlin instrumentation labels May 6, 2025

mcculls changed the title ~~WIP: Rework Kotlin coroutines instrumentation around coroutine context~~ Rework Kotlin coroutines instrumentation around coroutine context May 7, 2025

mcculls force-pushed the mcculls/rework-kotlin-instrumentation branch 4 times, most recently from a77d8bd to 06b989f Compare May 7, 2025 15:04

mcculls removed the tag: do not merge Do not merge changes label May 7, 2025

mcculls marked this pull request as ready for review May 7, 2025 16:13

mcculls requested review from a team as code owners May 7, 2025 16:13

mcculls requested review from amarziali and PerfectSlayer May 7, 2025 16:13

mcculls added tag: no release notes Changes to exclude from release notes and removed tag: no release notes Changes to exclude from release notes labels May 7, 2025

mcculls force-pushed the mcculls/rework-kotlin-instrumentation branch 3 times, most recently from 48b5305 to 37ba1f6 Compare May 11, 2025 18:23

Rework Kotlin coroutines instrumentation around coroutine context

91445ae

As an added benefit, this approach does not require field injection

mcculls force-pushed the mcculls/rework-kotlin-instrumentation branch from 37ba1f6 to 91445ae Compare May 11, 2025 20:24

mcculls mentioned this pull request May 11, 2025

Cleanup Zio fiber instrumentation to avoid repeated activation of continuation #8798

Merged

amarziali approved these changes May 12, 2025

View reviewed changes

PerfectSlayer reviewed May 12, 2025

View reviewed changes

...og/trace/instrumentation/kotlin/coroutines/AbstractKotlinCoroutineInstrumentationTest.groovy Show resolved Hide resolved

mcculls merged commit 4738946 into master May 12, 2025
590 of 591 checks passed

mcculls deleted the mcculls/rework-kotlin-instrumentation branch May 12, 2025 09:29

github-actions bot added this to the 1.50.0 milestone May 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rework Kotlin coroutines instrumentation around coroutine context #8774

Rework Kotlin coroutines instrumentation around coroutine context #8774

Uh oh!

mcculls commented May 6, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 6, 2025 •

edited

Loading

Uh oh!

amarziali left a comment

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rework Kotlin coroutines instrumentation around coroutine context #8774

Rework Kotlin coroutines instrumentation around coroutine context #8774

Uh oh!

Conversation

mcculls commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

amarziali left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcculls commented May 6, 2025 •

edited

Loading

pr-commenter bot commented May 6, 2025 •

edited

Loading