Update censys.io, Closes #439#2096
Conversation
* added book icon * Fix md Signed-off-by: 0ssigeno <[email protected]> * Adjusted toasts (intelowlproject#2015) * adjusted toast * fix test --------- Co-authored-by: Daniele Rosetti <[email protected]> * updated flare-floss to v3 * updated flare-floss to v3.0.1 * fixed the doc about contribute * fixed unresponsive scan page form with large files * changed social button popover trigger * fixed order problem in visualizer tabs * fixed PR template * fixed PR template * Frontend - HealthCheck and Pull buttons (intelowlproject#2025) * added healthcheck and pull button * test draft * frontend tests * adjusted toasts * added rule to short variable names (intelowlproject#2026) * added rule to short variable names * fix * fixed rules * fixed rules * added todo comment * fix * added note * fixed short var names * fixes * fix * fix * fix * comment * intelowlproject#1990 DNS0 /names Analyzer (intelowlproject#2024) * First not tested prototype * Added _monkeypatch() skeleton * Fixed a wrong condition in parameter validation * Added config() function to prepare data for request * Added hasattr() check for attributes * First successful run - Added check on existence of API parameter - Added authorization header - Added checks on parameters existence * Fix _api_name parameter check * Fixed parameter configuration * Removed unused if statement * Added support for UNIX timestamps and relative dates * Renamed dns0 service to separate services * Completed checks for not set parameters * Added checks of existence for all parameters * Added default Accept format header * Simplified regex to match single relative date * Fixed date check by forcing format * Modified _api_key to be non required * Made _api_key optional to support 10 free queries/day * Added /rrsets API * Moved files into separate dns0/ folder * Created module structure and base file * Extracted a common Mixin and applied to subclasses * Resolved performance issue flagged by deepsource * Added absolute import path * Substituted dateutils with dateparser and updated requirements * Updated Usage.md * Added explanation in Advanced-Usage.md * Fix dump Signed-off-by: 0ssigeno <[email protected]> * Black Signed-off-by: 0ssigeno <[email protected]> * Added analyzer config migrations * Added new analyzers to FREE_TO_USE_ANALYZERS playbook * Removed dns query type check * Removed unused variable 'query' * Added explicit default value in 'limit' parameter * Added DNS0 api docs reference * Solved deepsource warnings * Added custom user agent * Removed DNS0_rrsets_data * Solved deepsource warning * Added example JSON data * Generalization of ids Signed-off-by: 0ssigeno <[email protected]> * Fix dumpplugin migration generation Signed-off-by: 0ssigeno <[email protected]> * Excluding id Signed-off-by: 0ssigeno <[email protected]> * Excluding id in migration Signed-off-by: 0ssigeno <[email protected]> * Added default parameter in test * Added link to the DNS0 doc in plugin description * Fix pipeline test #1 * Fix pipeline test n2 * Fix pipeline test n3 * Fix pipeline test n4 * Fix pipeline test n5 * Fix pipeline test n6 * Fix pipeline test n7 * Fix pipeline test n8 * Fix pipeline test n9 * Fix pipeline test n10 * Fix pipeline test n11 * Fix pipeline test n12 * Fix pipeline test n13 * Fix pipeline test n14 * Fix pipeline test n15 * Fix pipeline test n16 * Fix pipeline test n17 * Removed unused q parameter --------- Signed-off-by: 0ssigeno <[email protected]> Co-authored-by: 0ssigeno <[email protected]> * Bump actions/setup-python from 4.7.0 to 5.0.0 (intelowlproject#2022) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.7.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4.7.0...v5.0.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google-cloud-webrisk from 1.12.0 to 1.13.0 in /requirements (intelowlproject#2018) Bumps [google-cloud-webrisk](https://github.com/googleapis/google-cloud-python) from 1.12.0 to 1.13.0. - [Release notes](https://github.com/googleapis/google-cloud-python/releases) - [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md) - [Commits](googleapis/google-cloud-python@google-cloud-webrisk-v1.12.0...google-cloud-webrisk-v1.13.0) --- updated-dependencies: - dependency-name: google-cloud-webrisk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Corner case migration Signed-off-by: 0ssigeno <[email protected]> * More logs Signed-off-by: 0ssigeno <[email protected]> * Fix Signed-off-by: 0ssigeno <[email protected]> * incremented logs in most used apis * Elastic bi (intelowlproject#2036) * Elastic bi Signed-off-by: 0ssigeno <[email protected]> * Elastic bi + fix elastic package Signed-off-by: 0ssigeno <[email protected]> * Update api_app/queryset.py Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * Fix Signed-off-by: 0ssigeno <[email protected]> * Index template Signed-off-by: 0ssigeno <[email protected]> * Fixes Signed-off-by: 0ssigeno <[email protected]> --------- Signed-off-by: 0ssigeno <[email protected]> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * changelog + docs + bump + django doctor + deepsource * Bump github/codeql-action from 2 to 3 (intelowlproject#2033) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump authlib from 1.2.0 to 1.3.0 in /requirements (intelowlproject#2031) Bumps [authlib](https://github.com/lepture/authlib) from 1.2.0 to 1.3.0. - [Release notes](https://github.com/lepture/authlib/releases) - [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst) - [Commits](authlib/authlib@v1.2.0...v1.3.0) --- updated-dependencies: - dependency-name: authlib dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed permissions and log files for ClamAV * Added IP and subdomain support for DNS0_rrsets_data (intelowlproject#2042) * Added IP support for DNS0_rrsets_data analyzer * Added include_subdomain parameter * Typo * Restore original state * Added alter migration to add a new supported type and new parameter * fix deepsource --------- Signed-off-by: 0ssigeno <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Daniele Rosetti <[email protected]> Co-authored-by: 0ssigeno <[email protected]> Co-authored-by: Martina Carella <[email protected]> Co-authored-by: Daniele Rosetti <[email protected]> Co-authored-by: fgibertoni <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simone Berni <[email protected]> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
* Using correct pipeline to manage visualizers (intelowlproject#2044) * Using correct pipeline to manage visualizers Signed-off-by: 0ssigeno <[email protected]> * Fix tests Signed-off-by: 0ssigeno <[email protected]> --------- Signed-off-by: 0ssigeno <[email protected]> * tweaks to docs * added exception catcher for Cymru analyzer * fixed bug in info column in the plugin section * fixed bug in multiple observable analysis (intelowlproject#2047) * fixed bug in multiple observable analysis: all observable had the type of the first one * linter * restore IP regex to accept also IPv6 * removed env in the toolbar (intelowlproject#2050) * removed env in the toolbar * linter * Dashboard improv: add button to filter for the user's org only (intelowlproject#2028) * frontend : add switch for user's org Signed-off-by: Shivam Purohit <[email protected]> * backend:add endpoint for the req Signed-off-by: Shivam Purohit <[email protected]> * initial backend url Signed-off-by: Shivam Purohit <[email protected]> * improve frontend logic Signed-off-by: Shivam Purohit <[email protected]> * add user org filter Signed-off-by: Shivam Purohit <[email protected]> * fix : try except block Signed-off-by: Shivam Purohit <[email protected]> * remove redundant url endpoints Signed-off-by: Shivam Purohit <[email protected]> * remove unnecessary commented code Signed-off-by: Shivam Purohit <[email protected]> * add hide component logic Signed-off-by: Shivam Purohit <[email protected]> * rename props Signed-off-by: Shivam Purohit <[email protected]> * reduce redundant code via functions Signed-off-by: Shivam Purohit <[email protected]> * convert get_org_members to static Signed-off-by: Shivam Purohit <[email protected]> * remove logger changes Signed-off-by: Shivam Purohit <[email protected]> * remove comments and rename variables Signed-off-by: Shivam Purohit <[email protected]> * remove console statement Signed-off-by: Shivam Purohit <[email protected]> * refactor get_org_members Signed-off-by: Shivam Purohit <[email protected]> * fix:prettier Signed-off-by: Shivam Purohit <[email protected]> --------- Signed-off-by: Shivam Purohit <[email protected]> * adjusted README * Added check for path Signed-off-by: 0ssigeno <[email protected]> * Fix corner case for multiple playbook Signed-off-by: 0ssigeno <[email protected]> * Fix pointers Signed-off-by: 0ssigeno <[email protected]> * Backend always decide the classification Signed-off-by: 0ssigeno <[email protected]> * Job bi (intelowlproject#2052) * job bi Signed-off-by: 0ssigeno <[email protected]> * More Signed-off-by: 0ssigeno <[email protected]> * Job bi Signed-off-by: 0ssigeno <[email protected]> * Migration Signed-off-by: 0ssigeno <[email protected]> * Fix Signed-off-by: 0ssigeno <[email protected]> * Fix Signed-off-by: 0ssigeno <[email protected]> * Added playbook in the elastic template Signed-off-by: 0ssigeno <[email protected]> * Fix Signed-off-by: 0ssigeno <[email protected]> * Missin migrations Signed-off-by: 0ssigeno <[email protected]> --------- Signed-off-by: 0ssigeno <[email protected]> * Bump pillow from 10.0.1 to 10.2.0 in /requirements (intelowlproject#2055) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump checkdmarc from 4.8.4 to 5.2.7 in /requirements (intelowlproject#2054) Bumps [checkdmarc](https://github.com/domainaware/checkdmarc) from 4.8.4 to 5.2.7. - [Changelog](https://github.com/domainaware/checkdmarc/blob/master/CHANGELOG.md) - [Commits](https://github.com/domainaware/checkdmarc/commits) --- updated-dependencies: - dependency-name: checkdmarc dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump quark-engine from 23.9.1 to 23.12.1 in /requirements (intelowlproject#2045) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 23.9.1 to 23.12.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](ev-flow/quark-engine@v23.9.1...v23.12.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump flake8 from 6.1.0 to 7.0.0 in /requirements (intelowlproject#2056) Bumps [flake8](https://github.com/pycqa/flake8) from 6.1.0 to 7.0.0. - [Commits](PyCQA/flake8@6.1.0...7.0.0) --- updated-dependencies: - dependency-name: flake8 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Frontend - fixed runtime config bugs (intelowlproject#2064) * fixed runtime config bugs * restored old comment --------- Co-authored-by: Daniele Rosetti <[email protected]> * updated frontend dependencies * Refactoring of start.py script. Closes intelowlproject#1899 and intelowlproject#1866 (intelowlproject#2060) * Removed initialize.sh root execution * Remove sudo command from installation * Added python3-venv installation * Dropped support for docker compose V1 * Added redirection to stderr * Dropped support for docker compose V1 * Fixed download directory of script * Changed shebang for better compatibility * First not finished bash script * Formatting fix * Added parameter checks * Added other parameters check * Fixed wrongly used check * Removed start() function * Used exact string matching for parameters * Added mandatory argument parsing * Moved declaration of array inside of logical function * Added main parameter case * Added help function * Added default version to help * Completed help function * Added argument parsing * Added help option * Fixed subtle error with bash logic values * Most of the work done * Working docker cmd * Added project directory specification * Removed unused function * Improved error message * Removed reduntand if condition * Added checks for empty strings * Removed pycharm warnings * Fixed IntelOwl version * Fixed some bugs * Fixed wrong for index * Fixed git checkout * Removed wrongly placed comment * Removed python pre-requirements depencency * Added execution of initialize.sh from start * Entering venv * Removed unused echo * Added check for docker group * Updated docs to use new startup script * Fixed typo in doc * Changed generic code blocks to bash * Add curl dependency * Add curl dependency * Re added pre-requirements for compatibility * Removed old occurrences of start.py * Added test docker file for integrations * Added manual usage to doc * Removed python3 and venv dependency * Updated major release informations * Removed python and venv dependency for start * Removed python and venv dependency for start from docs * Made code-review-doctor happy * Improved installation doc * Added absolute path for application_restart * Updated docs * added deprecation notice --------- Co-authored-by: Matteo Lodi <[email protected]> * Fixed creating a playbook with an existing tag (intelowlproject#2057) * changed playbook serializer * deepsource --------- Co-authored-by: Matteo Lodi <[email protected]> * bump and changelog --------- Signed-off-by: 0ssigeno <[email protected]> Signed-off-by: Shivam Purohit <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Simone Berni <[email protected]> Co-authored-by: Daniele Rosetti <[email protected]> Co-authored-by: Daniele Rosetti <[email protected]> Co-authored-by: Shivam Purohit <[email protected]> Co-authored-by: 0ssigeno <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Martina Carella <[email protected]> Co-authored-by: fgibertoni <[email protected]>
Update Censys.io analyzer to v2
|
@mlodic made required changes. I guess we are done. However I do have one small doubt. In the |
|
yes, that Before merging the PR, could you please add the example of the results in the MockUpResponse? That is something that I recently add to the PR request template but that does not appear in this PR template cause that is not in |
|
About the MockUpResponse: There's an issue wherein the json mockup gets formatted by the |
Signed-off-by: 0ssigeno <[email protected]>
|
Ignore the previous comment, I just had to store the mock json in a variable; all good |
Closes #439
Description
Upgraded the censys api endpoint to v2
Type of change
Please delete options that are not relevant.
Checklist
develop_monkeypatch()was used in its class to apply the necessary decorators.dumpplugincommand and added it in the project as a data migration. ("How to share a plugin with the community")test_files.zipand you added the default tests for that mimetype in test_classes.py.FREE_TO_USE_ANALYZERSplaybook by following this guide.Black,Flake,Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.testsfolder). All the tests (new and old ones) gave 0 errors.Important Rules
Raw Json:
{'code': 200, 'status': 'OK', 'result': {'ip': '190.121.56.10', 'services': [], 'location': {'continent': 'South America', 'country': 'Chile', 'country_code': 'CL', 'city': 'Osorno', 'postal_code': '5290000', 'timezone': 'America/Santiago', 'province': 'Los Lagos Region', 'coordinates': {'latitude': -40.57395, 'longitude': -73.13348}}, 'location_updated_at': '2024-01-27T14:52:11.775086600Z', 'autonomous_system': {'asn': 14117, 'description': 'Telefonica del Sur S.A.', 'bgp_prefix': '190.121.56.0/21', 'name': 'Telefonica del Sur S.A.', 'country_code': 'CL'}, 'autonomous_system_updated_at': '2024-01-27T14:52:11.775086600Z', 'last_updated_at': '2023-04-30T00:04:14.886Z'}}