Track the source of installation

[mabdinur]

What Does This Do

• Submits instrumentation.source, injection.enabled, and injection.force ssi configuration to the instrumentation telemetry platform.
• Computes whether was used to instrument a process in the AgentBootstrap.

Motivation

Track how services are instrumented.

Additional Notes

Contributor Checklist

• Format the title according the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any usefull labels
• Don't use close, fix or any linking keywords when referencing an issue.
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, move, or deletion
• Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

[mabdinur]

Follow up: https://github.com/DataDog/dd-go/pull/187441

[github-actions]

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

• Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

[PerfectSlayer]

Please move to a dedicated fonction and gives it a meaningful name

[mcculls]

As part of that could you add a trySetProperty method to the SystemUtils utility class like this:

  public static void trySetProperty(String property, String value) {
    try {
      System.setProperty(property, value);
    } catch (SecurityException e) {
      // ignore
    }
  }

This is just in case the user has a strict security manager installed - if so then this catches the security exception early and avoids having any side-effect on startup.

Then your dedicated function might look something like:

private static void recordInstrumentationSource(String source) {
      SystemUtils.trySetProperty(LIB_INSTRUMENTATION_SOURCE_SYS_PROP, source);
}

and the above code would become:

    if (getConfig(LIB_INJECTION_ENABLED_ENV_VAR)) {
      recordInstrumentationSource("ssi");
    } else {
      recordInstrumentationSource("cmd_line");
    }

[PerfectSlayer]

Default should not be null. Consider using enum instead if the values have limited options

[mcculls]

If there's no default (i.e. null) then we usually omit an entry here and use the simpler method, i.e. configProvider.getString(SSI_INJECTION_ENABLED);

Enums are quite weighty in terms of class-loading, so I prefer to avoid them in the tracer unless they add lots of value.

[PerfectSlayer]

Are those config only "get" to leverage the side effect of config telemetry?

[mcculls]

Correct - if we want to make this even more explicit in the code we could introduce some reportConfig methods which just delegate to configProvider.getXYZ and always return void but IMHO the leading comment here is enough.

We could also call ConfigCollector.get().put(....) directly to report but we'd still need to fetch the config.

[pr-commenter]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	munir/track-source-of-instrumentation-ssi
git_commit_date	1749739285	1749742520
git_commit_sha	1a9af7b	1463cd9
release_version	1.50.0-SNAPSHOT~1a9af7b785	1.50.0-SNAPSHOT~1463cd91b5

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1749744746	1749744746
ci_job_id	979104046	979104046
ci_pipeline_id	67559389	67559389
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-ytwrhy75-project-304-concurrent-0-z0a0vqir 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-ytwrhy75-project-304-concurrent-0-z0a0vqir 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~1463cd91b5, baseline=1.50.0-SNAPSHOT~1a9af7b785

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.035 s) : 0, 1034736
Total [baseline] (10.477 s) : 0, 10476794
Agent [candidate] (1.023 s) : 0, 1023058
Total [candidate] (10.452 s) : 0, 10451813
section appsec
Agent [baseline] (1.172 s) : 0, 1172411
Total [baseline] (10.598 s) : 0, 10598054
Agent [candidate] (1.173 s) : 0, 1172665
Total [candidate] (10.625 s) : 0, 10624581
section iast
Agent [baseline] (1.161 s) : 0, 1161214
Total [baseline] (10.869 s) : 0, 10868944
Agent [candidate] (1.158 s) : 0, 1158325
Total [candidate] (10.848 s) : 0, 10848369
section profiling
Agent [baseline] (1.267 s) : 0, 1266860
Total [baseline] (10.929 s) : 0, 10929354
Agent [candidate] (1.266 s) : 0, 1266248
Total [candidate] (10.775 s) : 0, 10775075

Loading

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.035 s	-
Agent	appsec	1.172 s	137.676 ms (13.3%)
Agent	iast	1.161 s	126.478 ms (12.2%)
Agent	profiling	1.267 s	232.125 ms (22.4%)
Total	tracing	10.477 s	-
Total	appsec	10.598 s	121.26 ms (1.2%)
Total	iast	10.869 s	392.15 ms (3.7%)
Total	profiling	10.929 s	452.56 ms (4.3%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.023 s	-
Agent	appsec	1.173 s	149.607 ms (14.6%)
Agent	iast	1.158 s	135.267 ms (13.2%)
Agent	profiling	1.266 s	243.19 ms (23.8%)
Total	tracing	10.452 s	-
Total	appsec	10.625 s	172.768 ms (1.7%)
Total	iast	10.848 s	396.556 ms (3.8%)
Total	profiling	10.775 s	323.262 ms (3.1%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~1463cd91b5, baseline=1.50.0-SNAPSHOT~1a9af7b785

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.181 ms) : 0, 692181
BytebuddyAgent [candidate] (683.094 ms) : 0, 683094
GlobalTracer [baseline] (242.398 ms) : 0, 242398
GlobalTracer [candidate] (240.972 ms) : 0, 240972
AppSec [baseline] (57.352 ms) : 0, 57352
AppSec [candidate] (57.108 ms) : 0, 57108
Debugger [baseline] (6.194 ms) : 0, 6194
Debugger [candidate] (6.117 ms) : 0, 6117
Remote Config [baseline] (727.626 µs) : 0, 728
Remote Config [candidate] (748.803 µs) : 0, 749
Telemetry [baseline] (12.086 ms) : 0, 12086
Telemetry [candidate] (11.386 ms) : 0, 11386
section appsec
BytebuddyAgent [baseline] (703.719 ms) : 0, 703719
BytebuddyAgent [candidate] (703.87 ms) : 0, 703870
GlobalTracer [baseline] (234.178 ms) : 0, 234178
GlobalTracer [candidate] (234.057 ms) : 0, 234057
AppSec [baseline] (175.353 ms) : 0, 175353
AppSec [candidate] (175.502 ms) : 0, 175502
Debugger [baseline] (5.912 ms) : 0, 5912
Debugger [candidate] (5.971 ms) : 0, 5971
Remote Config [baseline] (608.061 µs) : 0, 608
Remote Config [candidate] (618.424 µs) : 0, 618
Telemetry [baseline] (7.36 ms) : 0, 7360
Telemetry [candidate] (7.349 ms) : 0, 7349
IAST [baseline] (21.833 ms) : 0, 21833
IAST [candidate] (21.832 ms) : 0, 21832
section iast
BytebuddyAgent [baseline] (810.144 ms) : 0, 810144
BytebuddyAgent [candidate] (807.757 ms) : 0, 807757
GlobalTracer [baseline] (231.971 ms) : 0, 231971
GlobalTracer [candidate] (232.05 ms) : 0, 232050
AppSec [baseline] (55.771 ms) : 0, 55771
AppSec [candidate] (52.925 ms) : 0, 52925
Debugger [baseline] (6.012 ms) : 0, 6012
Debugger [candidate] (6.013 ms) : 0, 6013
Remote Config [baseline] (598.345 µs) : 0, 598
Remote Config [candidate] (621.512 µs) : 0, 622
Telemetry [baseline] (7.918 ms) : 0, 7918
Telemetry [candidate] (7.95 ms) : 0, 7950
IAST [baseline] (24.98 ms) : 0, 24980
IAST [candidate] (27.247 ms) : 0, 27247
section profiling
BytebuddyAgent [baseline] (676.075 ms) : 0, 676075
BytebuddyAgent [candidate] (675.101 ms) : 0, 675101
GlobalTracer [baseline] (361.157 ms) : 0, 361157
GlobalTracer [candidate] (359.998 ms) : 0, 359998
AppSec [baseline] (61.4 ms) : 0, 61400
AppSec [candidate] (62.031 ms) : 0, 62031
Debugger [baseline] (6.028 ms) : 0, 6028
Debugger [candidate] (6.101 ms) : 0, 6101
Remote Config [baseline] (658.337 µs) : 0, 658
Remote Config [candidate] (639.873 µs) : 0, 640
Telemetry [baseline] (8.139 ms) : 0, 8139
Telemetry [candidate] (8.172 ms) : 0, 8172
ProfilingAgent [baseline] (102.54 ms) : 0, 102540
ProfilingAgent [candidate] (103.348 ms) : 0, 103348
Profiling [baseline] (102.566 ms) : 0, 102566
Profiling [candidate] (103.372 ms) : 0, 103372

Loading

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~1463cd91b5, baseline=1.50.0-SNAPSHOT~1a9af7b785

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.027 s) : 0, 1027390
Total [baseline] (8.535 s) : 0, 8535464
Agent [candidate] (1.023 s) : 0, 1023363
Total [candidate] (8.537 s) : 0, 8536949
section iast
Agent [baseline] (1.149 s) : 0, 1149352
Total [baseline] (9.181 s) : 0, 9181094
Agent [candidate] (1.151 s) : 0, 1150858
Total [candidate] (9.19 s) : 0, 9190058
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.147 s) : 0, 1147471
Total [baseline] (9.153 s) : 0, 9153073
Agent [candidate] (1.148 s) : 0, 1148462
Total [candidate] (9.128 s) : 0, 9128195
section iast_TELEMETRY_OFF
Agent [baseline] (1.148 s) : 0, 1148065
Total [baseline] (9.236 s) : 0, 9235815
Agent [candidate] (1.145 s) : 0, 1144659
Total [candidate] (9.163 s) : 0, 9163077

Loading

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.027 s	-
Agent	iast	1.149 s	121.963 ms (11.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.147 s	120.081 ms (11.7%)
Agent	iast_TELEMETRY_OFF	1.148 s	120.676 ms (11.7%)
Total	tracing	8.535 s	-
Total	iast	9.181 s	645.631 ms (7.6%)
Total	iast_HARDCODED_SECRET_DISABLED	9.153 s	617.609 ms (7.2%)
Total	iast_TELEMETRY_OFF	9.236 s	700.352 ms (8.2%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.023 s	-
Agent	iast	1.151 s	127.496 ms (12.5%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.148 s	125.099 ms (12.2%)
Agent	iast_TELEMETRY_OFF	1.145 s	121.297 ms (11.9%)
Total	tracing	8.537 s	-
Total	iast	9.19 s	653.11 ms (7.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.128 s	591.247 ms (6.9%)
Total	iast_TELEMETRY_OFF	9.163 s	626.129 ms (7.3%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~1463cd91b5, baseline=1.50.0-SNAPSHOT~1a9af7b785

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.638 ms) : 0, 686638
BytebuddyAgent [candidate] (682.757 ms) : 0, 682757
GlobalTracer [baseline] (241.184 ms) : 0, 241184
GlobalTracer [candidate] (240.48 ms) : 0, 240480
AppSec [baseline] (55.993 ms) : 0, 55993
AppSec [candidate] (56.855 ms) : 0, 56855
Debugger [baseline] (6.192 ms) : 0, 6192
Debugger [candidate] (6.164 ms) : 0, 6164
Remote Config [baseline] (736.536 µs) : 0, 737
Remote Config [candidate] (719.241 µs) : 0, 719
Telemetry [baseline] (12.978 ms) : 0, 12978
Telemetry [candidate] (12.853 ms) : 0, 12853
section iast
BytebuddyAgent [baseline] (801.745 ms) : 0, 801745
BytebuddyAgent [candidate] (803.011 ms) : 0, 803011
GlobalTracer [baseline] (230.629 ms) : 0, 230629
GlobalTracer [candidate] (230.715 ms) : 0, 230715
AppSec [baseline] (52.155 ms) : 0, 52155
AppSec [candidate] (50.537 ms) : 0, 50537
Debugger [baseline] (5.953 ms) : 0, 5953
Debugger [candidate] (5.972 ms) : 0, 5972
Remote Config [baseline] (608.336 µs) : 0, 608
Remote Config [candidate] (597.804 µs) : 0, 598
Telemetry [baseline] (7.9 ms) : 0, 7900
Telemetry [candidate] (7.903 ms) : 0, 7903
IAST [baseline] (26.87 ms) : 0, 26870
IAST [candidate] (27.874 ms) : 0, 27874
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (799.948 ms) : 0, 799948
BytebuddyAgent [candidate] (800.817 ms) : 0, 800817
GlobalTracer [baseline] (230.135 ms) : 0, 230135
GlobalTracer [candidate] (230.27 ms) : 0, 230270
AppSec [baseline] (52.352 ms) : 0, 52352
AppSec [candidate] (51.16 ms) : 0, 51160
Debugger [baseline] (5.967 ms) : 0, 5967
Debugger [candidate] (5.95 ms) : 0, 5950
Remote Config [baseline] (589.487 µs) : 0, 589
Remote Config [candidate] (581.462 µs) : 0, 581
Telemetry [baseline] (7.944 ms) : 0, 7944
Telemetry [candidate] (7.923 ms) : 0, 7923
IAST [baseline] (27.08 ms) : 0, 27080
IAST [candidate] (28.262 ms) : 0, 28262
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (799.716 ms) : 0, 799716
BytebuddyAgent [candidate] (797.546 ms) : 0, 797546
GlobalTracer [baseline] (231.123 ms) : 0, 231123
GlobalTracer [candidate] (230.818 ms) : 0, 230818
AppSec [baseline] (48.723 ms) : 0, 48723
AppSec [candidate] (49.264 ms) : 0, 49264
Debugger [baseline] (5.973 ms) : 0, 5973
Debugger [candidate] (5.933 ms) : 0, 5933
Remote Config [baseline] (578.114 µs) : 0, 578
Remote Config [candidate] (596.899 µs) : 0, 597
Telemetry [baseline] (7.848 ms) : 0, 7848
Telemetry [candidate] (7.815 ms) : 0, 7815
IAST [baseline] (30.543 ms) : 0, 30543
IAST [candidate] (29.236 ms) : 0, 29236

Loading

Load

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	munir/track-source-of-instrumentation-ssi
git_commit_date	1749739285	1749742520
git_commit_sha	1a9af7b	1463cd9
release_version	1.50.0-SNAPSHOT~1a9af7b785	1.50.0-SNAPSHOT~1463cd91b5

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1749744595	1749744595
ci_job_id	979104052	979104052
ci_pipeline_id	67559389	67559389
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-ytwrhy75-project-304-concurrent-1-39jrv3q4 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-ytwrhy75-project-304-concurrent-1-39jrv3q4 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~1463cd91b5, baseline=1.50.0-SNAPSHOT~1a9af7b785
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.527 s) : 15527000, 15527000
.   : milestone, 15527000,
appsec (14.907 s) : 14907000, 14907000
.   : milestone, 14907000,
iast (18.493 s) : 18493000, 18493000
.   : milestone, 18493000,
iast_GLOBAL (17.81 s) : 17810000, 17810000
.   : milestone, 17810000,
profiling (15.266 s) : 15266000, 15266000
.   : milestone, 15266000,
tracing (14.965 s) : 14965000, 14965000
.   : milestone, 14965000,
section candidate
no_agent (15.748 s) : 15748000, 15748000
.   : milestone, 15748000,
appsec (14.876 s) : 14876000, 14876000
.   : milestone, 14876000,
iast (18.661 s) : 18661000, 18661000
.   : milestone, 18661000,
iast_GLOBAL (18.014 s) : 18014000, 18014000
.   : milestone, 18014000,
profiling (15.322 s) : 15322000, 15322000
.   : milestone, 15322000,
tracing (14.901 s) : 14901000, 14901000
.   : milestone, 14901000,

Loading

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.527 s [15.527 s, 15.527 s]	-
appsec	14.907 s [14.907 s, 14.907 s]	-620.0 ms (-4.0%)
iast	18.493 s [18.493 s, 18.493 s]	2.966 s (19.1%)
iast_GLOBAL	17.81 s [17.81 s, 17.81 s]	2.283 s (14.7%)
profiling	15.266 s [15.266 s, 15.266 s]	-261.0 ms (-1.7%)
tracing	14.965 s [14.965 s, 14.965 s]	-562.0 ms (-3.6%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.748 s [15.748 s, 15.748 s]	-
appsec	14.876 s [14.876 s, 14.876 s]	-872.0 ms (-5.5%)
iast	18.661 s [18.661 s, 18.661 s]	2.913 s (18.5%)
iast_GLOBAL	18.014 s [18.014 s, 18.014 s]	2.266 s (14.4%)
profiling	15.322 s [15.322 s, 15.322 s]	-426.0 ms (-2.7%)
tracing	14.901 s [14.901 s, 14.901 s]	-847.0 ms (-5.4%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~1463cd91b5, baseline=1.50.0-SNAPSHOT~1a9af7b785
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.482 ms) : 1470, 1494
.   : milestone, 1482,
appsec (2.411 ms) : 2363, 2460
.   : milestone, 2411,
iast (2.196 ms) : 2135, 2257
.   : milestone, 2196,
iast_GLOBAL (2.241 ms) : 2178, 2303
.   : milestone, 2241,
profiling (2.024 ms) : 1975, 2072
.   : milestone, 2024,
tracing (2.006 ms) : 1959, 2054
.   : milestone, 2006,
section candidate
no_agent (1.484 ms) : 1473, 1496
.   : milestone, 1484,
appsec (2.406 ms) : 2358, 2455
.   : milestone, 2406,
iast (2.194 ms) : 2133, 2255
.   : milestone, 2194,
iast_GLOBAL (2.236 ms) : 2175, 2298
.   : milestone, 2236,
profiling (2.048 ms) : 1998, 2098
.   : milestone, 2048,
tracing (2.018 ms) : 1971, 2066
.   : milestone, 2018,

Loading

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.482 ms [1.47 ms, 1.494 ms]	-
appsec	2.411 ms [2.363 ms, 2.46 ms]	929.346 µs (62.7%)
iast	2.196 ms [2.135 ms, 2.257 ms]	714.039 µs (48.2%)
iast_GLOBAL	2.241 ms [2.178 ms, 2.303 ms]	758.655 µs (51.2%)
profiling	2.024 ms [1.975 ms, 2.072 ms]	541.581 µs (36.5%)
tracing	2.006 ms [1.959 ms, 2.054 ms]	524.32 µs (35.4%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.484 ms [1.473 ms, 1.496 ms]	-
appsec	2.406 ms [2.358 ms, 2.455 ms]	922.352 µs (62.1%)
iast	2.194 ms [2.133 ms, 2.255 ms]	709.927 µs (47.8%)
iast_GLOBAL	2.236 ms [2.175 ms, 2.298 ms]	752.134 µs (50.7%)
profiling	2.048 ms [1.998 ms, 2.098 ms]	564.16 µs (38.0%)
tracing	2.018 ms [1.971 ms, 2.066 ms]	534.359 µs (36.0%)