Nuclei analyzer by pranjalg1331 · Pull Request #2697 · intelowlproject/IntelOwl

pranjalg1331 · 2025-01-25T15:17:21Z

Description

Closes #1883

Sample Resonse-

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
[ x] New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

integrations/nuclei_analyzer/app.py

mlodic · 2025-01-27T10:47:01Z

what is the status of this PR?

pranjalg1331 · 2025-01-27T10:53:22Z

@mlodic, Sir the analyzer is working. I just have to add _monkeypatch(), and then It should be complete.

mlodic · 2025-01-27T10:54:32Z

thanks for the update please add a reviewer once you finish!

pranjalg1331 · 2025-01-27T18:48:17Z

@drosetti Please review.

mlodic · 2025-01-28T16:03:20Z

I'm reviewing it

mlodic

good work but this task is a little more complicated. Important changes must be made.

Plus, when a new docker analyzer is added, also this section must be updated:

https://github.com/intelowlproject/IntelOwl/blob/master/.github/dependabot.yml

api_app/analyzers_manager/migrations/0148_analyzer_config_nuclei.py

mlodic · 2025-01-28T16:06:17Z

api_app/analyzers_manager/migrations/0148_analyzer_config_nuclei.py

+        "base_path": "api_app.analyzers_manager.observable_analyzers",
+    },
+    "name": "Nuclei",
+    "description": "A fast and customisable vulnerability scanner powered by simple YAML-based templates",


add markdown link to the Nuclei webpage or docs. That would be rendered by the frontend

api_app/analyzers_manager/migrations/0148_analyzer_config_nuclei.py

integrations/nuclei_analyzer/Dockerfile

mlodic · 2025-01-28T16:14:09Z

integrations/nuclei_analyzer/compose-tests.yml

+    build:
+      context: ../integrations/nuclei_analyzer
+      dockerfile: Dockerfile
+    image: pranjalg1310/nuclei-analyzer:1.2.2


switch to test instance here, see other docker analyzers

this has not been solved

integrations/nuclei_analyzer/app.py

api_app/analyzers_manager/observable_analyzers/nuclei.py

integrations/nuclei_analyzer/app.py

mlodic · 2025-01-28T16:39:36Z

integrations/nuclei_analyzer/Dockerfile

+    mv nuclei /usr/local/bin/ && \
+    chmod +x /usr/local/bin/nuclei && \
+    rm nuclei_3.3.7_linux_amd64.zip && \
+    nuclei -update-template-dir /opt/nuclei-api/nuclei-templates -update-templates && \


you do this to update the template project, right? That's a good thing to do.
However, I did not see any implementation of a way to periodically update them. I expect an implementation of a simple cronjob that daily would check and get updates of these templates. In the pcap_analyzer integration we already added a cronjob, you can take example from there.

@mlodic I checked the implementation of cron job in pcap_analyzer but could not understand how we are calling these two files - crontab and update_signature. I also tried running pcap analyzer with updated timing to update every minute but could not see any logs being created inside the container.
Could you please guide me more on how to setup cronjob for docker based analyzer and how are they being called?

mlodic · 2025-01-28T16:51:28Z

integrations/nuclei_analyzer/Dockerfile

+    wget \
+    unzip && \
+    wget --progress=dot:giga https://github.com/projectdiscovery/nuclei/releases/download/v3.3.7/nuclei_3.3.7_linux_amd64.zip && \
+    unzip nuclei_3.3.7_linux_amd64.zip && \


this approach has 2 problems:

first, a custom docker image is really needed? they provide one https://docs.projectdiscovery.io/tools/nuclei/install.

if it is really needed, cannot this be the base of this custom docker image instead of python? That would help maintain a cleaner image plus we would be able to track Nuclei version updates with the dependabot. If you download the binary like this, we wouldn't be able to be notified by the dependabot.

dependabot still missing

I have added the Dependabot config. Although I did not know how I could test it, so please take a look.

integrations/nuclei_analyzer/app.py

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 · 2025-02-02T19:17:01Z

I have successfully implemented the flask-shell2http library in my Flask API and used the Nuclei Docker image as the base for my Dockerfile. This has significantly simplified the code.

Currently, I haven't updated the Dockerfile version in compose-test.yaml to a test version yet. Could you review it before I proceed?

api_app/analyzers_manager/observable_analyzers/nuclei.py

api_app/analyzers_manager/migrations/0148_analyzer_config_nuclei.py

mlodic · 2025-02-05T10:52:59Z

integrations/nuclei_analyzer/compose-tests.yml

+    build:
+      context: ../integrations/nuclei_analyzer
+      dockerfile: Dockerfile
+    image: pranjalg1310/nuclei-analyzer:1.2.2


this has not been solved

mlodic · 2025-02-05T10:53:11Z

api_app/analyzers_manager/migrations/0148_analyzer_config_nuclei.py

+        "base_path": "api_app.analyzers_manager.observable_analyzers",
+    },
+    "name": "Nuclei",
+    "description": "A fast and customisable vulnerability scanner powered by simple YAML-based templates",


mlodic · 2025-02-05T10:53:21Z

integrations/nuclei_analyzer/compose.yml

+
+services:
+  nuclei_analyzer:
+    image: pranjalg1310/nuclei-analyzer:1.2.3


mlodic · 2025-02-05T10:59:24Z

integrations/nuclei_analyzer/entrypoint.sh

+while [ ! -d "$TEMPLATES_DIR" ] || [ -z "$(ls -A $TEMPLATES_DIR)" ]; do
+    echo "Templates not found or empty, retrying update in 10 seconds..."
+    nuclei -update-template-dir $TEMPLATES_DIR -update-templates
+    sleep 10
+done


whis is a potential infinite loop, do not use while like this. In case the templates are not found, the entrypoint should exit with a custom error

updated the code to exit if templates are not downloaded successfully.

mlodic · 2025-02-05T11:00:13Z

integrations/nuclei_analyzer/entrypoint.sh

+done
+
+echo "Templates downloaded successfully. Starting Flask API..."
+exec gunicorn -b 0.0.0.0:4008 --timeout 120 --access-logfile - "app:app"


use all the options we used in the other entrypoint.sh you can found

Included gunicorn logging same as Suricata.

integrations/nuclei_analyzer/Dockerfile

mlodic · 2025-02-05T11:01:20Z

integrations/nuclei_analyzer/Dockerfile

+    wget \
+    unzip && \
+    wget --progress=dot:giga https://github.com/projectdiscovery/nuclei/releases/download/v3.3.7/nuclei_3.3.7_linux_amd64.zip && \
+    unzip nuclei_3.3.7_linux_amd64.zip && \


dependabot still missing

api_app/analyzers_manager/observable_analyzers/nuclei.py

mlodic · 2025-02-05T11:05:24Z

Currently, I haven't updated the Dockerfile version in compose-test.yaml to a test version yet. Could you review it before I proceed?

You can already do this during development because you would use the local built from the compose-tests-yml that would overwrite the build from compose.yml

Signed-off-by: pranjalg1331 <[email protected]>

.github/dependabot.yml

Signed-off-by: pranjalg1331 <[email protected]>

mlodic

this task has been challenging. Last steps and we are ready to merge

mlodic · 2025-02-13T09:15:37Z

api_app/analyzers_manager/observable_analyzers/nuclei.py

+            if template_dir in VALID_TEMPLATE_CATEGORIES:
+                args.extend(["-t", template_dir])
+            else:
+                print(f"Skipping invalid template directory: {template_dir}")


this should be a warning log and you should add it in the report like this self.report.errors.append(warning). Remember to reference the analyzed observable in the log

mlodic · 2025-02-13T09:16:56Z

integrations/nuclei_analyzer/Dockerfile

+RUN adduser -D -h /home/${USER} ${USER}
+
+# Install required packages using apk and clean cache in the same layer
+RUN apk add --no-cache python3 py3-pip \


please select a specific python version here. The more recent the better.

mlodic · 2025-02-13T09:19:02Z

integrations/nuclei_analyzer/app.py

+            try:
+                json_objects.append(json.loads(line))
+            except json.JSONDecodeError:
+                print(f"Skipping non-JSON line: {line}")


log.warning

mlodic · 2025-02-13T09:19:33Z

integrations/nuclei_analyzer/compose.yml

+
+services:
+  nuclei_analyzer:
+    image: pranjalg1310/nuclei-analyzer:{REACT_APP_INTELOWL_VERSION}


use the image from intelowlproject repo

mlodic · 2025-02-13T09:19:49Z

integrations/nuclei_analyzer/compose-tests.yml

+    build:
+      context: ../integrations/nuclei_analyzer
+      dockerfile: Dockerfile
+    image: pranjalg1310/nuclei-analyzer:test


intelowlproject repo

mlodic · 2025-02-13T09:23:33Z

integrations/nuclei_analyzer/Dockerfile

+# Expose the API port
+EXPOSE 4008
+
+ENTRYPOINT ["/entrypoint.sh"]


could you please add an HEALTHCHECK here as an additional check?

You can check the Dockerfile_nginx as an example. You can check whether the gunicorn app is up or not after a predefined period that should be the expected time of execution of the entrypoint.sh script

For the health check, I have added an extra endpoint to the API which is being called from the dockerfile.

Signed-off-by: pranjalg1331 <[email protected]>

mlodic · 2025-02-17T11:36:54Z

great!

* Install Kanvas workflow * initial commit Signed-off-by: pranjalg1331 <[email protected]> * Dockerfile Signed-off-by: pranjalg1331 <[email protected]> * synchronous nuclei api Signed-off-by: pranjalg1331 <[email protected]> * removing unnecessary file Signed-off-by: pranjalg1331 <[email protected]> * dockerfile Signed-off-by: pranjalg1331 <[email protected]> * made api async Signed-off-by: pranjalg1331 <[email protected]> * error resolved Signed-off-by: pranjalg1331 <[email protected]> * async working Signed-off-by: pranjalg1331 <[email protected]> * another Signed-off-by: pranjalg1331 <[email protected]> * deepsource errors Signed-off-by: pranjalg1331 <[email protected]> * gunicorn version fixed Signed-off-by: pranjalg1331 <[email protected]> * shell2http architecture Signed-off-by: pranjalg1331 <[email protected]> * deep sorce errors resolved Signed-off-by: pranjalg1331 <[email protected]> * nuclei file update Signed-off-by: pranjalg1331 <[email protected]> * new version for dockerfile Signed-off-by: pranjalg1331 <[email protected]> * migration file changes Signed-off-by: pranjalg1331 <[email protected]> * error corected Signed-off-by: pranjalg1331 <[email protected]> * error resolved Signed-off-by: pranjalg1331 <[email protected]> * dockerfile cleanup code Signed-off-by: pranjalg1331 <[email protected]> * dependabot added + dockerfile test version Signed-off-by: pranjalg1331 <[email protected]> * dependabot for python dependencies Signed-off-by: pranjalg1331 <[email protected]> * healthcheck for api Signed-off-by: pranjalg1331 <[email protected]> --------- Signed-off-by: pranjalg1331 <[email protected]> Co-authored-by: meshery-dev[bot] <132387951+meshery-dev[bot]@users.noreply.github.com>

github-advanced-security bot found potential problems Jan 25, 2025

View reviewed changes

integrations/nuclei_analyzer/app.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jan 26, 2025

View reviewed changes

integrations/nuclei_analyzer/app.py Fixed Show fixed Hide fixed

pranjalg1331 force-pushed the NucleiAnalyzer branch from 8e3021e to 1b474e2 Compare January 27, 2025 18:25

pranjalg1331 changed the title ~~Nuclei analyzer Draft~~ Nuclei analyzer Jan 27, 2025

mlodic requested changes Jan 28, 2025

View reviewed changes

github-advanced-security bot found potential problems Jan 31, 2025

View reviewed changes

integrations/nuclei_analyzer/app.py Fixed Show fixed Hide fixed

meshery-dev bot and others added 16 commits February 2, 2025 20:38

Install Kanvas workflow

149592d

initial commit

c3633e4

Signed-off-by: pranjalg1331 <[email protected]>

Dockerfile

c80d5cb

Signed-off-by: pranjalg1331 <[email protected]>

synchronous nuclei api

83de7f6

Signed-off-by: pranjalg1331 <[email protected]>

removing unnecessary file

8b7a8fe

Signed-off-by: pranjalg1331 <[email protected]>

dockerfile

e6dc793

Signed-off-by: pranjalg1331 <[email protected]>

made api async

8d32be0

Signed-off-by: pranjalg1331 <[email protected]>

error resolved

2b1741a

Signed-off-by: pranjalg1331 <[email protected]>

async working

ed73fe6

Signed-off-by: pranjalg1331 <[email protected]>

another

b0a0bae

Signed-off-by: pranjalg1331 <[email protected]>

deepsource errors

03be7ae

Signed-off-by: pranjalg1331 <[email protected]>

gunicorn version fixed

ff07e79

Signed-off-by: pranjalg1331 <[email protected]>

shell2http architecture

06a48e1

Signed-off-by: pranjalg1331 <[email protected]>

deep sorce errors resolved

6955600

Signed-off-by: pranjalg1331 <[email protected]>

nuclei file update

acaa6c7

Signed-off-by: pranjalg1331 <[email protected]>

new version for dockerfile

eed403c

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 force-pushed the NucleiAnalyzer branch from a8eac2b to eed403c Compare February 2, 2025 17:56

pranjalg1331 added 2 commits February 2, 2025 23:57

migration file changes

63b2ba7

Signed-off-by: pranjalg1331 <[email protected]>

error corected

99724d8

Signed-off-by: pranjalg1331 <[email protected]>

error resolved

c377723

Signed-off-by: pranjalg1331 <[email protected]>

mlodic requested changes Feb 5, 2025

View reviewed changes

pranjalg1331 added 2 commits February 6, 2025 20:50

dockerfile cleanup code

b70f2fd

Signed-off-by: pranjalg1331 <[email protected]>

dependabot added + dockerfile test version

5dfff61

Signed-off-by: pranjalg1331 <[email protected]>

mlodic mentioned this pull request Feb 7, 2025

Markdown not supported in Plugin Descriptions #2717

Open

mlodic reviewed Feb 7, 2025

View reviewed changes

.github/dependabot.yml Show resolved Hide resolved

dependabot for python dependencies

2e742a7

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 requested a review from mlodic February 12, 2025 08:49

mlodic requested changes Feb 13, 2025

View reviewed changes

healthcheck for api

7862ced

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 requested a review from mlodic February 14, 2025 20:38

mlodic merged commit f3ac867 into intelowlproject:develop Feb 17, 2025
10 checks passed

Uh oh!

Conversation

pranjalg1331 commented Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Important Rules

Uh oh!

Uh oh!

Uh oh!

mlodic commented Jan 27, 2025

Uh oh!

pranjalg1331 commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlodic commented Jan 27, 2025

Uh oh!

pranjalg1331 commented Jan 27, 2025

Uh oh!

mlodic commented Jan 28, 2025

Uh oh!

mlodic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pranjalg1331 commented Feb 2, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pranjalg1331 Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlodic commented Feb 5, 2025

pranjalg1331 commented Jan 25, 2025 •

edited

Loading

pranjalg1331 commented Jan 27, 2025 •

edited

Loading

mlodic left a comment •

edited

Loading

pranjalg1331 Feb 6, 2025 •

edited

Loading