#1990 DNS0 /names Analyzer by fgibertoni · Pull Request #2024 · intelowlproject/IntelOwl

fgibertoni · 2023-12-12T10:40:42Z

Description

Added a new analyzer that uses DNS0 /names API to query for current and historical DNS data.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

- Added check on existence of API parameter - Added authorization header - Added checks on parameters existence

api_app/analyzers_manager/observable_analyzers/dns0_names.py

mlodic

I left some comments, keep up the good work :)

api_app/analyzers_manager/migrations/0054_analyzer_config_dns0_rrsets_name.py

codecov · 2023-12-18T16:01:58Z

Codecov Report

Merging #2024 (6491c89) into develop (aa8820f) will increase coverage by 8.99%.
Report is 1569 commits behind head on develop.
The diff coverage is 74.63%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2024      +/-   ##
===========================================
+ Coverage    66.75%   75.74%   +8.99%     
===========================================
  Files           95      471     +376     
  Lines         3706    14192   +10486     
  Branches       519     1482     +963     
===========================================
+ Hits          2474    10750    +8276     
- Misses         941     2853    +1912     
- Partials       291      589     +298

Files	Coverage Δ
api_app/analyzers_manager/admin.py	`100.00% <100.00%> (ø)`
api_app/analyzers_manager/apps.py	`100.00% <100.00%> (ø)`
api_app/analyzers_manager/constants.py	`100.00% <100.00%> (ø)`
api_app/analyzers_manager/exceptions.py	`100.00% <100.00%> (ø)`
...app/analyzers_manager/file_analyzers/boxjs_scan.py	`100.00% <100.00%> (ø)`
...pi_app/analyzers_manager/file_analyzers/onenote.py	`100.00% <100.00%> (ø)`
...pi_app/analyzers_manager/file_analyzers/peframe.py	`82.35% <100.00%> (ø)`
...i_app/analyzers_manager/file_analyzers/rtf_info.py	`48.64% <100.00%> (ø)`
...s_manager/file_analyzers/xlm_macro_deobfuscator.py	`62.50% <100.00%> (ø)`
api_app/analyzers_manager/filters.py	`100.00% <100.00%> (ø)`
... and 236 more

... and 281 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e4a481...6491c89. Read the comment docs.

mlodic · 2023-12-18T16:17:07Z

Great! This is your first contribution to the project! Kudos :)

* added book icon * Fix md Signed-off-by: 0ssigeno <[email protected]> * Adjusted toasts (#2015) * adjusted toast * fix test --------- Co-authored-by: Daniele Rosetti <[email protected]> * updated flare-floss to v3 * updated flare-floss to v3.0.1 * fixed the doc about contribute * fixed unresponsive scan page form with large files * changed social button popover trigger * fixed order problem in visualizer tabs * fixed PR template * fixed PR template * Frontend - HealthCheck and Pull buttons (#2025) * added healthcheck and pull button * test draft * frontend tests * adjusted toasts * added rule to short variable names (#2026) * added rule to short variable names * fix * fixed rules * fixed rules * added todo comment * fix * added note * fixed short var names * fixes * fix * fix * fix * comment * #1990 DNS0 /names Analyzer (#2024) * First not tested prototype * Added _monkeypatch() skeleton * Fixed a wrong condition in parameter validation * Added config() function to prepare data for request * Added hasattr() check for attributes * First successful run - Added check on existence of API parameter - Added authorization header - Added checks on parameters existence * Fix _api_name parameter check * Fixed parameter configuration * Removed unused if statement * Added support for UNIX timestamps and relative dates * Renamed dns0 service to separate services * Completed checks for not set parameters * Added checks of existence for all parameters * Added default Accept format header * Simplified regex to match single relative date * Fixed date check by forcing format * Modified _api_key to be non required * Made _api_key optional to support 10 free queries/day * Added /rrsets API * Moved files into separate dns0/ folder * Created module structure and base file * Extracted a common Mixin and applied to subclasses * Resolved performance issue flagged by deepsource * Added absolute import path * Substituted dateutils with dateparser and updated requirements * Updated Usage.md * Added explanation in Advanced-Usage.md * Fix dump Signed-off-by: 0ssigeno <[email protected]> * Black Signed-off-by: 0ssigeno <[email protected]> * Added analyzer config migrations * Added new analyzers to FREE_TO_USE_ANALYZERS playbook * Removed dns query type check * Removed unused variable 'query' * Added explicit default value in 'limit' parameter * Added DNS0 api docs reference * Solved deepsource warnings * Added custom user agent * Removed DNS0_rrsets_data * Solved deepsource warning * Added example JSON data * Generalization of ids Signed-off-by: 0ssigeno <[email protected]> * Fix dumpplugin migration generation Signed-off-by: 0ssigeno <[email protected]> * Excluding id Signed-off-by: 0ssigeno <[email protected]> * Excluding id in migration Signed-off-by: 0ssigeno <[email protected]> * Added default parameter in test * Added link to the DNS0 doc in plugin description * Fix pipeline test #1 * Fix pipeline test n2 * Fix pipeline test n3 * Fix pipeline test n4 * Fix pipeline test n5 * Fix pipeline test n6 * Fix pipeline test n7 * Fix pipeline test n8 * Fix pipeline test n9 * Fix pipeline test n10 * Fix pipeline test n11 * Fix pipeline test n12 * Fix pipeline test n13 * Fix pipeline test n14 * Fix pipeline test n15 * Fix pipeline test n16 * Fix pipeline test n17 * Removed unused q parameter --------- Signed-off-by: 0ssigeno <[email protected]> Co-authored-by: 0ssigeno <[email protected]> * Bump actions/setup-python from 4.7.0 to 5.0.0 (#2022) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.7.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4.7.0...v5.0.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google-cloud-webrisk from 1.12.0 to 1.13.0 in /requirements (#2018) Bumps [google-cloud-webrisk](https://github.com/googleapis/google-cloud-python) from 1.12.0 to 1.13.0. - [Release notes](https://github.com/googleapis/google-cloud-python/releases) - [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md) - [Commits](googleapis/google-cloud-python@google-cloud-webrisk-v1.12.0...google-cloud-webrisk-v1.13.0) --- updated-dependencies: - dependency-name: google-cloud-webrisk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Corner case migration Signed-off-by: 0ssigeno <[email protected]> * More logs Signed-off-by: 0ssigeno <[email protected]> * Fix Signed-off-by: 0ssigeno <[email protected]> * incremented logs in most used apis * Elastic bi (#2036) * Elastic bi Signed-off-by: 0ssigeno <[email protected]> * Elastic bi + fix elastic package Signed-off-by: 0ssigeno <[email protected]> * Update api_app/queryset.py Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * Fix Signed-off-by: 0ssigeno <[email protected]> * Index template Signed-off-by: 0ssigeno <[email protected]> * Fixes Signed-off-by: 0ssigeno <[email protected]> --------- Signed-off-by: 0ssigeno <[email protected]> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * changelog + docs + bump + django doctor + deepsource * Bump github/codeql-action from 2 to 3 (#2033) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump authlib from 1.2.0 to 1.3.0 in /requirements (#2031) Bumps [authlib](https://github.com/lepture/authlib) from 1.2.0 to 1.3.0. - [Release notes](https://github.com/lepture/authlib/releases) - [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst) - [Commits](authlib/authlib@v1.2.0...v1.3.0) --- updated-dependencies: - dependency-name: authlib dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed permissions and log files for ClamAV * Added IP and subdomain support for DNS0_rrsets_data (#2042) * Added IP support for DNS0_rrsets_data analyzer * Added include_subdomain parameter * Typo * Restore original state * Added alter migration to add a new supported type and new parameter * fix deepsource --------- Signed-off-by: 0ssigeno <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Daniele Rosetti <[email protected]> Co-authored-by: 0ssigeno <[email protected]> Co-authored-by: Martina Carella <[email protected]> Co-authored-by: Daniele Rosetti <[email protected]> Co-authored-by: fgibertoni <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simone Berni <[email protected]> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>

fgibertoni added 15 commits December 7, 2023 12:28

First not tested prototype

9c002ef

Added _monkeypatch() skeleton

30d03f1

Fixed a wrong condition in parameter validation

92a0282

Added config() function to prepare data for request

ce90196

Added hasattr() check for attributes

d536ff1

First successful run

3b687c8

- Added check on existence of API parameter - Added authorization header - Added checks on parameters existence

Fix _api_name parameter check

0981e58

Fixed parameter configuration

bef154a

Removed unused if statement

be24eb5

Added support for UNIX timestamps and relative dates

b4551ea

Renamed dns0 service to separate services

5f43674

Completed checks for not set parameters

321c92d

Added checks of existence for all parameters

d8fbb94

Added default Accept format header

20a2ffd

Simplified regex to match single relative date

97a48ae

fgibertoni changed the title ~~[#1990](https://github.com/intelowlproject/IntelOwl/issues/1990) DNS0 /names Analyzer~~ #1990 DNS0 /names Analyzer Dec 12, 2023

mlodic reviewed Dec 12, 2023

View reviewed changes

api_app/analyzers_manager/observable_analyzers/dns0_names.py Outdated Show resolved Hide resolved

mlodic reviewed Dec 12, 2023

View reviewed changes

api_app/analyzers_manager/observable_analyzers/dns0_names.py Outdated Show resolved Hide resolved

mlodic reviewed Dec 12, 2023

View reviewed changes

api_app/analyzers_manager/observable_analyzers/dns0_names.py Outdated Show resolved Hide resolved

mlodic reviewed Dec 12, 2023

View reviewed changes

fgibertoni added 10 commits December 12, 2023 17:44

Fixed date check by forcing format

c84aa72

Modified _api_key to be non required

67b7b2c

Made _api_key optional to support 10 free queries/day

34da240

Added /rrsets API

a3bf42c

Moved files into separate dns0/ folder

4dfd5ae

Created module structure and base file

03069ff

Extracted a common Mixin and applied to subclasses

6394b3b

Resolved performance issue flagged by deepsource

22292df

Added absolute import path

5eeeb7d

Substituted dateutils with dateparser and updated requirements

88e86bc

mlodic reviewed Dec 15, 2023

View reviewed changes

api_app/analyzers_manager/migrations/0054_analyzer_config_dns0_rrsets_name.py Outdated Show resolved Hide resolved

fgibertoni added 20 commits December 15, 2023 11:15

Added default parameter in test

768b674

Added link to the DNS0 doc in plugin description

235ac81

Fix pipeline test #1

e8b1939

Fix pipeline test n2

d373cbf

Fix pipeline test n3

3eb9f0c

Fix pipeline test n4

9e7d7f0

Fix pipeline test n5

7aa4ae6

Fix pipeline test n6

89d672c

Fix pipeline test n7

55598dd

Fix pipeline test n8

93ac631

Fix pipeline test n9

fb0c350

Fix pipeline test n10

71a5662

Fix pipeline test n11

bf2d5a2

Fix pipeline test n12

87ba6cc

Fix pipeline test n13

cb4813e

Fix pipeline test n14

511cb12

Fix pipeline test n15

3eeafc7

Fix pipeline test n16

29e52e8

Fix pipeline test n17

7b7a0df

Removed unused q parameter

6491c89

fgibertoni requested a review from mlodic December 18, 2023 16:04

mlodic linked an issue Dec 18, 2023 that may be closed by this pull request

[Analyzer] DNS0 PassiveDNS #1990

Closed

mlodic merged commit 407f053 into develop Dec 18, 2023

mlodic deleted the dns0-api-integration branch December 18, 2023 16:17

fgibertoni restored the dns0-api-integration branch December 27, 2023 11:22

fgibertoni deleted the dns0-api-integration branch January 16, 2024 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

#1990 DNS0 /names Analyzer#2024

#1990 DNS0 /names Analyzer#2024
mlodic merged 67 commits intodevelopfrom
dns0-api-integration

fgibertoni commented Dec 12, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlodic left a comment

Uh oh!

Uh oh!

codecov bot commented Dec 18, 2023

Uh oh!

mlodic commented Dec 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

fgibertoni commented Dec 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Important Rules

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Dec 18, 2023

Codecov Report

Uh oh!

mlodic commented Dec 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fgibertoni commented Dec 12, 2023 •

edited

Loading