Update appsec rules from Remote Config by uurien · Pull Request #2654 · DataDog/dd-trace-js

uurien · 2023-01-04T15:38:17Z

What does this PR do?

This PR adds the availability to update more waf configurations from Remote Configuration. It can update the rules based on new rules file coming in new RC ASM event or it can override previously configured rules or add exclusion with data received in ASM_DD RC event.

As we had a bit overengineered waf gateway code, this PR adds a new and simpler approach to do the same stuff.

TODO:

codecov · 2023-01-04T15:41:39Z

Codecov Report

Merging #2654 (41cc9b2) into master (256271d) will decrease coverage by 0.16%.
The diff coverage is 94.63%.

@@            Coverage Diff             @@
##           master    #2654      +/-   ##
==========================================
- Coverage   87.35%   87.19%   -0.16%     
==========================================
  Files         319      317       -2     
  Lines       11534    11512      -22     
  Branches       33       33              
==========================================
- Hits        10075    10038      -37     
- Misses       1459     1474      +15

Impacted Files	Coverage Δ
packages/dd-trace/src/appsec/addresses.js	`100.00% <ø> (ø)`
.../dd-trace/src/appsec/remote_config/capabilities.js	`100.00% <ø> (ø)`
packages/dd-trace/src/appsec/rule_manager.js	`88.99% <88.76%> (-11.01%)`	⬇️
...ges/dd-trace/src/appsec/waf/waf_context_wrapper.js	`92.00% <92.00%> (ø)`
packages/dd-trace/src/appsec/waf/index.js	`93.33% <93.33%> (ø)`
packages/datadog-plugin-http/src/server.js	`94.11% <100.00%> (ø)`
packages/datadog-plugin-http2/src/server.js	`96.15% <100.00%> (ø)`
packages/dd-trace/src/appsec/channels.js	`100.00% <100.00%> (ø)`
packages/dd-trace/src/appsec/index.js	`100.00% <100.00%> (ø)`
.../dd-trace/src/appsec/remote_config/apply_states.js	`100.00% <100.00%> (ø)`
... and 5 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pr-commenter · 2023-01-04T16:19:34Z

Benchmarks

Comparing candidate commit cb0d967 in PR branch ugaitz/asm_dd-rules-update with baseline commit 256271d in branch master.

Found 2 performance improvements and 2 performance regressions! Performance is the same for 662 metrics, 42 unstable metrics.

scenario:appsec-appsec-enabled-with-attacks-14

🟥 cpu_usage_percentage [+7.994%; +9.708%]
🟩 execution_time [-45.367ms; -36.880ms] or [-10.359%; -8.421%]

scenario:appsec-appsec-enabled-with-attacks-16

🟥 cpu_usage_percentage [+8.648%; +9.889%]
🟩 execution_time [-46.419ms; -39.687ms] or [-10.342%; -8.842%]

github-actions · 2023-01-19T15:51:58Z

Overall package size

Self size: 3.98 MB
Deduped: 64.2 MB
No deduping: 64.25 MB

Dependency sizes

name	version	self size	total size
@datadog/native-iast-taint-tracking	1.4.0	14.89 MB	14.9 MB
@datadog/pprof	2.2.0	13.71 MB	14.59 MB
@datadog/native-appsec	3.1.0	13.31 MB	13.32 MB
@datadog/native-metrics	1.6.0	7.88 MB	7.89 MB
protobufjs	7.1.2	2.76 MB	6.55 MB
@datadog/native-iast-rewriter	2.0.1	2.09 MB	2.1 MB
opentracing	0.14.7	194.81 kB	194.81 kB
semver	7.3.8	88.2 kB	118.6 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
lru-cache	7.14.0	74.95 kB	74.95 kB
ipaddr.js	2.0.1	59.52 kB	59.52 kB
ignore	5.2.0	48.87 kB	48.87 kB
import-in-the-middle	1.3.5	34.34 kB	38.81 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
retry	0.10.1	27.44 kB	27.44 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
limiter	1.1.5	23.17 kB	23.17 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
node-abort-controller	3.0.1	14.33 kB	14.33 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
diagnostics_channel	1.1.0	7.07 kB	7.07 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

This reverts commit a0c2514.

* Add rules update and refactor whole waf * move appsec channels file * Add changes to work with libddwaf update method and support more RC properties * update waf bindings to 3.1.0 * simplify reporter * remove unnecesary virtual addresses * cleanup outdated comment * simplify incomingHttpStartHandler * simplify waf_manager.js * add pre update hook to RC manager * update rc manager * batch all waf update RC subscriptions * put apply_states codes in a constant file * inline function * add update to waf manager * completely rework RC waf updates * renamed a few things in tests * fix dc path * fix webpack not parsing class fields * lazy load everything * fix unapply configs not being emitted --------- Co-authored-by: simon-id <[email protected]> Co-authored-by: Julio Gonzalez <[email protected]>

uurien added do-not-merge/WIP semver-patch asm-waf labels Jan 4, 2023

uurien force-pushed the ugaitz/asm_dd-rules-update branch from e488c36 to 57c1e98 Compare January 4, 2023 16:05

uurien force-pushed the ugaitz/asm_dd-rules-update branch 2 times, most recently from 15cca4c to 139d70a Compare January 4, 2023 17:04

uurien removed the do-not-merge/WIP label Jan 9, 2023

uurien force-pushed the ugaitz/asm_dd-rules-update branch from 139d70a to 4219955 Compare January 9, 2023 12:00

uurien marked this pull request as ready for review January 9, 2023 12:00

uurien requested a review from a team as a code owner January 9, 2023 12:00

Qard previously approved these changes Jan 9, 2023

View reviewed changes

uurien dismissed Qard’s stale review via b3b450c January 10, 2023 15:02

uurien force-pushed the ugaitz/asm_dd-rules-update branch from 4219955 to b3b450c Compare January 10, 2023 15:02

Qard previously approved these changes Jan 11, 2023

View reviewed changes

simon-id suggested changes Jan 19, 2023

View reviewed changes

Comment thread packages/dd-trace/src/appsec/remote_config/index.js Outdated

Comment thread packages/dd-trace/src/appsec/remote_config/index.js Outdated

uurien dismissed Qard’s stale review via ce06bdd January 19, 2023 15:51

Qard previously approved these changes Jan 19, 2023

View reviewed changes

uurien requested a review from simon-id January 20, 2023 13:18

simon-id reviewed Jan 20, 2023

View reviewed changes

Comment thread packages/dd-trace/src/appsec/remote_config/index.js Outdated

uurien dismissed Qard’s stale review via b5b4c05 January 20, 2023 16:37

Qard previously approved these changes Jan 20, 2023

View reviewed changes

simon-id reviewed Jan 22, 2023

View reviewed changes

Comment thread packages/dd-trace/src/appsec/rule_manager.js Outdated

simon-id reviewed Jan 23, 2023

View reviewed changes

Comment thread packages/dd-trace/test/appsec/remote_config/index.spec.js Outdated

simon-id reviewed Jan 23, 2023

View reviewed changes

Comment thread packages/dd-trace/test/appsec/remote_config/index.spec.js Outdated

simon-id reviewed Jan 23, 2023

View reviewed changes

Comment thread packages/dd-trace/test/appsec/remote_config/index.spec.js Outdated

uurien dismissed Qard’s stale review via c5ffb29 January 23, 2023 14:53

uurien force-pushed the ugaitz/asm_dd-rules-update branch from 7425c3c to 585fe54 Compare January 24, 2023 09:04

hoolioh and others added 7 commits April 24, 2023 12:22

Bring back symbol primitive to event name.

3b97991

Fix nonsense require and comments from previous commit.

eaceeae

Fix linting issues.

8280a7a

Merge branch 'master' into ugaitz/asm_dd-rules-update

3280bb8

Update manager.js

acee59a

Update manager.js

9b19c64

fix webpack not parsing class fields

c37723b

simon-id marked this pull request as ready for review April 25, 2023 15:40

hoolioh self-requested a review April 25, 2023 15:45

Merge branch 'master' into ugaitz/asm_dd-rules-update

91985c9

hoolioh previously approved these changes Apr 26, 2023

View reviewed changes

lazy load everything

561d2ee

simon-id dismissed hoolioh’s stale review via 561d2ee April 26, 2023 08:08

simon-id added 6 commits April 26, 2023 13:20

Update .eslintrc.json

5e8afb3

cleanup

e5d641d

cleanup

ba4c4d7

even more cleanup

cb0d967

fix unapply configs not being emitted

918a2b3

cleanup

41cc9b2

simon-id approved these changes Apr 26, 2023

View reviewed changes

hoolioh self-requested a review April 26, 2023 13:50

hoolioh approved these changes Apr 26, 2023

View reviewed changes

simon-id merged commit a0c2514 into master Apr 26, 2023

tlhunter added a commit that referenced this pull request Apr 27, 2023

Revert "Update appsec rules from Remote Config (#2654)"

b096cf6

This reverts commit a0c2514.

tlhunter mentioned this pull request May 2, 2023

Revert "Introduce service naming framework (#2941)" #3093

Merged

simon-id deleted the ugaitz/asm_dd-rules-update branch January 8, 2024 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update appsec rules from Remote Config#2654

Update appsec rules from Remote Config#2654
simon-id merged 60 commits into
masterfrom
ugaitz/asm_dd-rules-update

uurien commented Jan 4, 2023 •

edited by simon-id

Loading

Uh oh!

codecov Bot commented Jan 4, 2023 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jan 4, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jan 19, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

uurien commented Jan 4, 2023 • edited by simon-id Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

TODO:

Uh oh!

codecov Bot commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter Bot commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:appsec-appsec-enabled-with-attacks-14

scenario:appsec-appsec-enabled-with-attacks-16

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jan 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Dependency sizes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

uurien commented Jan 4, 2023 •

edited by simon-id

Loading

codecov Bot commented Jan 4, 2023 •

edited

Loading

pr-commenter Bot commented Jan 4, 2023 •

edited

Loading

github-actions Bot commented Jan 19, 2023 •

edited

Loading