HTTP response schema collection and data classification #8938
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sezen-datadog
force-pushed
the
sezen.leblay/APPSEC-57259-extract-schema-spring
branch
from
f3bdd40 to
7c044fd
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 61 metrics, 10 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~9ac9a163f8, baseline=1.50.0-SNAPSHOT~d2d38c9d44
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.024 s) : 0, 1023628
Total [baseline] (10.543 s) : 0, 10543149
Agent [candidate] (1.026 s) : 0, 1026263
Total [candidate] (10.516 s) : 0, 10516474
section appsec
Agent [baseline] (1.172 s) : 0, 1172036
Total [baseline] (10.673 s) : 0, 10673313
Agent [candidate] (1.168 s) : 0, 1167784
Total [candidate] (10.669 s) : 0, 10668944
section iast
Agent [baseline] (1.155 s) : 0, 1155278
Total [baseline] (10.869 s) : 0, 10868987
Agent [candidate] (1.151 s) : 0, 1150772
Total [candidate] (10.921 s) : 0, 10920541
section profiling
Agent [baseline] (1.274 s) : 0, 1273627
Total [baseline] (10.881 s) : 0, 10880876
Agent [candidate] (1.266 s) : 0, 1265911
Total [candidate] (10.886 s) : 0, 10885548
gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~9ac9a163f8, baseline=1.50.0-SNAPSHOT~d2d38c9d44
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (683.704 ms) : 0, 683704
BytebuddyAgent [candidate] (684.286 ms) : 0, 684286
GlobalTracer [baseline] (240.627 ms) : 0, 240627
GlobalTracer [candidate] (241.017 ms) : 0, 241017
AppSec [baseline] (56.58 ms) : 0, 56580
AppSec [candidate] (56.509 ms) : 0, 56509
Debugger [baseline] (6.187 ms) : 0, 6187
Debugger [candidate] (6.292 ms) : 0, 6292
Remote Config [baseline] (746.724 µs) : 0, 747
Remote Config [candidate] (709.402 µs) : 0, 709
Telemetry [baseline] (12.153 ms) : 0, 12153
Telemetry [candidate] (13.873 ms) : 0, 13873
section appsec
BytebuddyAgent [baseline] (707.117 ms) : 0, 707117
BytebuddyAgent [candidate] (703.33 ms) : 0, 703330
GlobalTracer [baseline] (238.86 ms) : 0, 238860
GlobalTracer [candidate] (238.36 ms) : 0, 238360
AppSec [baseline] (177.357 ms) : 0, 177357
AppSec [candidate] (177.421 ms) : 0, 177421
Debugger [baseline] (5.997 ms) : 0, 5997
Debugger [candidate] (6.012 ms) : 0, 6012
Remote Config [baseline] (642.719 µs) : 0, 643
Remote Config [candidate] (634.006 µs) : 0, 634
Telemetry [baseline] (7.411 ms) : 0, 7411
Telemetry [candidate] (7.398 ms) : 0, 7398
IAST [baseline] (21.802 ms) : 0, 21802
IAST [candidate] (21.888 ms) : 0, 21888
section iast
BytebuddyAgent [baseline] (804.71 ms) : 0, 804710
BytebuddyAgent [candidate] (802.338 ms) : 0, 802338
GlobalTracer [baseline] (232.205 ms) : 0, 232205
GlobalTracer [candidate] (230.509 ms) : 0, 230509
AppSec [baseline] (52.969 ms) : 0, 52969
AppSec [candidate] (55.464 ms) : 0, 55464
Debugger [baseline] (6.001 ms) : 0, 6001
Debugger [candidate] (6.025 ms) : 0, 6025
Remote Config [baseline] (600.59 µs) : 0, 601
Remote Config [candidate] (591.66 µs) : 0, 592
Telemetry [baseline] (8.029 ms) : 0, 8029
Telemetry [candidate] (7.886 ms) : 0, 7886
IAST [baseline] (27.133 ms) : 0, 27133
IAST [candidate] (24.397 ms) : 0, 24397
section profiling
BytebuddyAgent [baseline] (679.25 ms) : 0, 679250
BytebuddyAgent [candidate] (675.382 ms) : 0, 675382
GlobalTracer [baseline] (362.305 ms) : 0, 362305
GlobalTracer [candidate] (359.023 ms) : 0, 359023
AppSec [baseline] (61.797 ms) : 0, 61797
AppSec [candidate] (62.029 ms) : 0, 62029
Debugger [baseline] (6.119 ms) : 0, 6119
Debugger [candidate] (6.135 ms) : 0, 6135
Remote Config [baseline] (656.352 µs) : 0, 656
Remote Config [candidate] (649.503 µs) : 0, 650
Telemetry [baseline] (8.217 ms) : 0, 8217
Telemetry [candidate] (8.178 ms) : 0, 8178
ProfilingAgent [baseline] (104.11 ms) : 0, 104110
ProfilingAgent [candidate] (103.86 ms) : 0, 103860
Profiling [baseline] (104.135 ms) : 0, 104135
Profiling [candidate] (103.884 ms) : 0, 103884
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~9ac9a163f8, baseline=1.50.0-SNAPSHOT~d2d38c9d44
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.022 s) : 0, 1022307
Total [baseline] (8.522 s) : 0, 8521529
Agent [candidate] (1.037 s) : 0, 1037347
Total [candidate] (8.577 s) : 0, 8576595
section iast
Agent [baseline] (1.148 s) : 0, 1147808
Total [baseline] (9.176 s) : 0, 9175728
Agent [candidate] (1.157 s) : 0, 1157119
Total [candidate] (9.214 s) : 0, 9214245
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.151 s) : 0, 1151362
Total [baseline] (9.155 s) : 0, 9155377
Agent [candidate] (1.158 s) : 0, 1158226
Total [candidate] (9.164 s) : 0, 9163783
section iast_TELEMETRY_OFF
Agent [baseline] (1.143 s) : 0, 1142760
Total [baseline] (9.263 s) : 0, 9262504
Agent [candidate] (1.149 s) : 0, 1148968
Total [candidate] (9.314 s) : 0, 9313716
gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~9ac9a163f8, baseline=1.50.0-SNAPSHOT~d2d38c9d44
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (683.092 ms) : 0, 683092
BytebuddyAgent [candidate] (692.758 ms) : 0, 692758
GlobalTracer [baseline] (240.087 ms) : 0, 240087
GlobalTracer [candidate] (243.269 ms) : 0, 243269
AppSec [baseline] (58.148 ms) : 0, 58148
AppSec [candidate] (57.967 ms) : 0, 57967
Debugger [baseline] (6.09 ms) : 0, 6090
Debugger [candidate] (6.368 ms) : 0, 6368
Remote Config [baseline] (727.819 µs) : 0, 728
Remote Config [candidate] (720.987 µs) : 0, 721
Telemetry [baseline] (10.51 ms) : 0, 10510
Telemetry [candidate] (12.346 ms) : 0, 12346
section iast
BytebuddyAgent [baseline] (801.039 ms) : 0, 801039
BytebuddyAgent [candidate] (807.469 ms) : 0, 807469
GlobalTracer [baseline] (229.698 ms) : 0, 229698
GlobalTracer [candidate] (231.701 ms) : 0, 231701
AppSec [baseline] (52.208 ms) : 0, 52208
AppSec [candidate] (52.834 ms) : 0, 52834
Debugger [baseline] (5.929 ms) : 0, 5929
Debugger [candidate] (5.969 ms) : 0, 5969
Remote Config [baseline] (599.642 µs) : 0, 600
Remote Config [candidate] (588.46 µs) : 0, 588
Telemetry [baseline] (7.869 ms) : 0, 7869
Telemetry [candidate] (7.977 ms) : 0, 7977
IAST [baseline] (26.937 ms) : 0, 26937
IAST [candidate] (27.042 ms) : 0, 27042
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (803.085 ms) : 0, 803085
BytebuddyAgent [candidate] (807.395 ms) : 0, 807395
GlobalTracer [baseline] (230.564 ms) : 0, 230564
GlobalTracer [candidate] (232.064 ms) : 0, 232064
AppSec [baseline] (53.388 ms) : 0, 53388
AppSec [candidate] (53.139 ms) : 0, 53139
Debugger [baseline] (5.988 ms) : 0, 5988
Debugger [candidate] (6.05 ms) : 0, 6050
Remote Config [baseline] (614.474 µs) : 0, 614
Remote Config [candidate] (611.509 µs) : 0, 612
Telemetry [baseline] (7.979 ms) : 0, 7979
Telemetry [candidate] (8.042 ms) : 0, 8042
IAST [baseline] (26.141 ms) : 0, 26141
IAST [candidate] (27.253 ms) : 0, 27253
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (796.226 ms) : 0, 796226
BytebuddyAgent [candidate] (799.914 ms) : 0, 799914
GlobalTracer [baseline] (229.775 ms) : 0, 229775
GlobalTracer [candidate] (230.991 ms) : 0, 230991
AppSec [baseline] (51.958 ms) : 0, 51958
AppSec [candidate] (53.37 ms) : 0, 53370
Debugger [baseline] (5.913 ms) : 0, 5913
Debugger [candidate] (6.075 ms) : 0, 6075
Remote Config [baseline] (591.019 µs) : 0, 591
Remote Config [candidate] (605.445 µs) : 0, 605
Telemetry [baseline] (7.744 ms) : 0, 7744
Telemetry [candidate] (7.983 ms) : 0, 7983
IAST [baseline] (27.045 ms) : 0, 27045
IAST [candidate] (26.536 ms) : 0, 26536
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 3 performance regressions! Performance is the same for 0 metrics, 9 unstable metrics.
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~9ac9a163f8, baseline=1.50.0-SNAPSHOT~d2d38c9d44
dateFormat X
axisFormat %s
section baseline
no_agent (1.473 ms) : 1462, 1485
. : milestone, 1473,
appsec (2.397 ms) : 2349, 2446
. : milestone, 2397,
iast (2.188 ms) : 2127, 2249
. : milestone, 2188,
iast_GLOBAL (2.232 ms) : 2171, 2294
. : milestone, 2232,
profiling (2.044 ms) : 1994, 2094
. : milestone, 2044,
tracing (1.999 ms) : 1951, 2046
. : milestone, 1999,
section candidate
no_agent (1.474 ms) : 1462, 1485
. : milestone, 1474,
appsec (2.39 ms) : 2342, 2438
. : milestone, 2390,
iast (2.179 ms) : 2118, 2241
. : milestone, 2179,
iast_GLOBAL (2.217 ms) : 2156, 2278
. : milestone, 2217,
profiling (2.014 ms) : 1966, 2063
. : milestone, 2014,
tracing (2.018 ms) : 1970, 2066
. : milestone, 2018,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~9ac9a163f8, baseline=1.50.0-SNAPSHOT~d2d38c9d44
dateFormat X
axisFormat %s
section baseline
no_agent (14.98 s) : 14980000, 14980000
. : milestone, 14980000,
appsec (14.881 s) : 14881000, 14881000
. : milestone, 14881000,
iast (18.861 s) : 18861000, 18861000
. : milestone, 18861000,
iast_GLOBAL (18.12 s) : 18120000, 18120000
. : milestone, 18120000,
profiling (15.422 s) : 15422000, 15422000
. : milestone, 15422000,
tracing (15.08 s) : 15080000, 15080000
. : milestone, 15080000,
section candidate
no_agent (15.065 s) : 15065000, 15065000
. : milestone, 15065000,
appsec (14.893 s) : 14893000, 14893000
. : milestone, 14893000,
iast (18.846 s) : 18846000, 18846000
. : milestone, 18846000,
iast_GLOBAL (17.791 s) : 17791000, 17791000
. : milestone, 17791000,
profiling (15.797 s) : 15797000, 15797000
. : milestone, 15797000,
tracing (14.73 s) : 14730000, 14730000
. : milestone, 14730000,
|
jandro996
reviewed
Jun 9, 2025
| @@ -627,6 +671,38 @@ private Flow<Void> onRequestBodyDone(RequestContext ctx_, StoredBodySupplier sup | |||
| } | |||
| } | |||
|
|
|||
| private Flow<Void> onResponseBodyDone(RequestContext ctx_, StoredBodySupplier supplier) { | |||
There was a problem hiding this comment.
I need to check the RFC properly but AFAIK response body raw doesn't applies to schema collection
jandro996
reviewed
Jun 10, 2025
Comment on lines
+149
to
+160
| if (action instanceof Flow.Action.RequestBlockingAction) { | ||
| Flow.Action.RequestBlockingAction rba = (Flow.Action.RequestBlockingAction) action; | ||
| BlockResponseFunction brf = reqCtx.getBlockResponseFunction(); | ||
| if (brf != null) { | ||
| brf.tryCommitBlockingResponse( | ||
| reqCtx.getTraceSegment(), | ||
| rba.getStatusCode(), | ||
| rba.getBlockingContentType(), | ||
| rba.getExtraHeaders()); | ||
| } | ||
| throw new BlockingException("Blocked response (for HttpMessageConverter/write)"); | ||
| } |
There was a problem hiding this comment.
Do we need to do this here? we do this in the request to block if it's necessary.
jandro996
reviewed
Jun 10, 2025
| @Advice.OnMethodEnter(suppress = Throwable.class) | ||
| public static void before( | ||
| @Advice.Argument(0) final Object obj, @ActiveRequestContext RequestContext reqCtx) { | ||
| if (obj == null) { |
There was a problem hiding this comment.
As we don't want to collect the response if the request is blocked, maybe it's a good point to check something like AppSecRequestContext#isWafBlocked (only works for waf but we can add another flag for RASP)
jandro996
reviewed
Jun 10, 2025
| if (subInfo == null || subInfo.isEmpty()) { | ||
| return NoopFlow.INSTANCE; | ||
| } | ||
| Object converted = ObjectIntrospection.convert(obj, ctx, () -> {}); |
There was a problem hiding this comment.
We need to change the ObjectInstrospection#conver to be able to use it for response schema collection
Limits
Currently most libraries enforce a set of limits before serialising addresses into ddwaf_object. The default global object limits are the following:
Maximum string length: 4096 bytes
Maximum container depth: 20 levels
Maximum container size: 256 nodes
The schema extraction algorithm has a different set of limits, which are lower than the limits mentioned above:
Maximum container depth: 18 levels
Maximum array size: 10 nodes
Maximum record size: 255 nodes
When serialising addresses to ddwaf_object which aren’t used for anything other than schema extraction, the library may use the schema extraction limits, rather than the global object limits.
https://docs.google.com/document/d/1965kNw_1CScNM15GgLZ0jvMhFH2kLpDYGztw8YeOfjM/edit?tab=t.0
jandro996
reviewed
Jun 10, 2025
There was a problem hiding this comment.
I think that we only need the RESPONSE_BODY_CONVERTED
jandro996
reviewed
Jun 10, 2025
There was a problem hiding this comment.
Same here, I think that we are ok just with RESPONSE_BODY_CONVERTED_ID
jandro996
reviewed
Jun 10, 2025
| @@ -98,6 +98,7 @@ public class AppSecRequestContext implements DataBundle, Closeable { | |||
| private String inferredClientIp; | |||
|
|
|||
| private volatile StoredBodySupplier storedRequestBodySupplier; | |||
| private volatile StoredBodySupplier storedResponseBodySupplier; | |||
There was a problem hiding this comment.
I think we don't need this, as we can pass the response object via callback directly
jandro996
reviewed
Jun 10, 2025
| @@ -106,6 +107,8 @@ public class AppSecRequestContext implements DataBundle, Closeable { | |||
| private boolean rawReqBodyPublished; | |||
| private boolean convertedReqBodyPublished; | |||
| private boolean respDataPublished; | |||
| private boolean rawRespBodyPublished; | |||
sezen-datadog
force-pushed
the
sezen.leblay/APPSEC-57259-extract-schema-spring
branch
2 times, most recently
from
ade8110 to
5aa9177
Compare
sezen-datadog
changed the base branch from
master
to
malvarez/vertx-response-extraction
sezen-datadog
force-pushed
the
sezen.leblay/APPSEC-57259-extract-schema-spring
branch
from
5aa9177 to
5000116
Compare
sezen-datadog
force-pushed
the
sezen.leblay/APPSEC-57259-extract-schema-spring
branch
from
b2f2eb6 to
81198be
Compare
sezen-datadog
force-pushed
the
sezen.leblay/APPSEC-57259-extract-schema-spring
branch
from
64ee2ba to
a82275e
Compare
Signed-off-by: sezen.leblay <[email protected]>
sezen-datadog
force-pushed
the
sezen.leblay/APPSEC-57259-extract-schema-spring
branch
from
c65407e to
9ac9a16
Compare
manuel-alvarez-alvarez
force-pushed
the
malvarez/vertx-response-extraction
branch
3 times, most recently
from
414f1ea to
e6d0da9
Compare
Base automatically changed from
malvarez/vertx-response-extraction
to
malvarez/http-route-play
June 17, 2025 17:22
manuel-alvarez-alvarez
force-pushed
the
malvarez/http-route-play
branch
from
a9e3f59 to
ad782b6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: APPSEC-57259