Skip to content

Malshare ingestor#2804

Merged
fgibertoni merged 12 commits intointelowlproject:developfrom
pranjalg1331:malshare_ingestor
Mar 28, 2025
Merged

Malshare ingestor#2804
fgibertoni merged 12 commits intointelowlproject:developfrom
pranjalg1331:malshare_ingestor

Conversation

@pranjalg1331
Copy link
Contributor

@pranjalg1331 pranjalg1331 commented Mar 22, 2025 •
edited by spoiicy
Loading

Closes #1399

Description

Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue).
  • [ x] New feature (non-breaking change which adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

  • [ x] I have read and understood the rules about how to Contribute to this project
  • [ x] The pull request is for the branch develop
  • [ x] A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
    • [x ] I strictly followed the documentation "How to create a Plugin"
    • Usage file was updated. A link to the PR to the docs repo has been added as a comment here.
    • Advanced-Usage was updated (in case the plugin provides additional optional configuration). A link to the PR to the docs repo has been added as a comment here.
    • [ x] I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
    • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
    • If you created a new analyzer and it is free (does not require any API key), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
    • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
    • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
    • [ x] If the plugin interacts with an external service, I have created an attribute called precisely url that contains this information. This is required for Health Checks.
    • [ x] If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
    • [ x] I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
  • I have inserted the copyright banner at the start of the file: # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.
  • If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
  • [ x] Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
  • I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
  • If the GUI has been modified:
    • I have a provided a screenshot of the result in the PR.
    • I have created new frontend tests for the new component or updated existing ones.
  • After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

  • If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
  • Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review by using GitHub's reviewing system detailed here.

@pranjalg1331
Copy link
Contributor Author

pranjalg1331 commented Mar 23, 2025

Screenshot from 2025-03-22 19-39-40
Screenshot from 2025-03-22 19-39-34

@pranjalg1331 pranjalg1331 marked this pull request as ready for review March 23, 2025 12:54
@pranjalg1331
Copy link
Contributor Author

pranjalg1331 commented Mar 23, 2025

Please review.

fgibertoni
fgibertoni requested changes Mar 24, 2025
View reviewed changes
api_app/ingestors_manager/ingestors/malshare.py Outdated
response = requests.get(download_url)
response.raise_for_status()
if not isinstance(response.content, bytes):
raise ValueError("VT downloaded file is not instance of bytes")
Copy link
Contributor

@fgibertoni fgibertoni Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VT ?

Copy link
Contributor Author

@pranjalg1331 pranjalg1331 Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, misstype. Updated.

api_app/ingestors_manager/ingestors/malshare.py Outdated
try:
logger.info(f"Downloading sample {h}")
download_url = f"{self.url}/api.php?api_key={self._api_key_name}&action=getfile&hash={h}"
response = requests.get(download_url)
Copy link
Contributor

@fgibertoni fgibertoni Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please use the requests.get params parameter for each of the request parameter ? This avoid longer f-string and I think it's more readable

Copy link
Contributor Author

@pranjalg1331 pranjalg1331 Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

api_app/ingestors_manager/ingestors/malshare.py Outdated

def run(self) -> Iterable[Any]:
req_url = f"{self.url}/api.php?api_key={self._api_key_name}&action=getlist"
result = requests.get(req_url)
Copy link
Contributor

@fgibertoni fgibertoni Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please use the requests.get params parameter for each of the request parameter ? This avoid longer f-string and I think it's more readable

@pranjalg1331 pranjalg1331 requested a review from fgibertoni March 25, 2025 12:35
@fgibertoni
Copy link
Contributor

fgibertoni commented Mar 25, 2025 •
edited
Loading

Please pull the latest changes from develop to fix the error in CI

@pranjalg1331
Copy link
Contributor Author

pranjalg1331 commented Mar 25, 2025

I have pulled the latest changes, but the test is still failing.

@fgibertoni
Copy link
Contributor

fgibertoni commented Mar 25, 2025 •
edited
Loading

I don't think you did it correctly. You need to have this commit in your branch

@pranjalg1331
Copy link
Contributor Author

pranjalg1331 commented Mar 25, 2025 •
edited
Loading

image

image
Changes from your commit are already being reflected in my branch.

@pranjalg1331
Copy link
Contributor Author

pranjalg1331 commented Mar 25, 2025

Should I push a commit to fix the Spamhaus error?

@fgibertoni
Copy link
Contributor

fgibertoni commented Mar 25, 2025 •
edited
Loading

Can you please try pulling again ? Should be solved now hopefully

@pranjalg1331
initial commit
fde1839 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
malshare analyzer
2648546 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
checkpoint
2b9bc3c 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
migration file
810e97f 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
malshare ingestor
943e339 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
remove comment
0f70c8e 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
correcting monkeypatch
bc49b75 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
test fix
e9e9095 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
review
d369b1a 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331
test fix
43e1dd7 
Signed-off-by: pranjalg1331 <[email protected]>
fgibertoni
fgibertoni requested changes Mar 25, 2025
View reviewed changes
api_app/ingestors_manager/ingestors/malshare.py Outdated
def download_sample(self, h):
try:
logger.info(f"Downloading sample {h}")
download_url = f"{self.url}/api.php"
Copy link
Contributor

@fgibertoni fgibertoni Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is duplicated, I think we can set it as class attribute

Copy link
Contributor Author

@pranjalg1331 pranjalg1331 Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created base_url class attribute.

Copy link
Contributor

@fgibertoni fgibertoni Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about having a class attribute called endpoint that has it's value to api.php.
Then, when making the request you combine self.url and self.endpoint

Copy link
Contributor Author

@pranjalg1331 pranjalg1331 Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it..Updated

@pranjalg1331
minor changes
1a57241 
Signed-off-by: pranjalg1331 <[email protected]>
fgibertoni
fgibertoni requested changes Mar 26, 2025
View reviewed changes
fgibertoni
fgibertoni reviewed Mar 26, 2025
View reviewed changes
api_app/ingestors_manager/ingestors/malshare.py Outdated

def download_sample(self, sample_hash: str) -> bytes:
try:
logger.info(f"Downloading sample {sample_hash}")
Copy link
Contributor

@fgibertoni fgibertoni Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also write a more meaningful log here (different from run), so we can follow the execution flow during debugging.

Copy link
Contributor Author

@pranjalg1331 pranjalg1331 Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@pranjalg1331
suggetions
e8b7fa6 
Signed-off-by: pranjalg1331 <[email protected]>
@pranjalg1331 pranjalg1331 requested a review from fgibertoni March 27, 2025 16:33
@fgibertoni fgibertoni merged commit 35d5911 into intelowlproject:develop Mar 28, 2025
10 checks passed
@spoiicy spoiicy mentioned this pull request Jun 30, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fgibertoni fgibertoni fgibertoni approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pranjalg1331 @fgibertoni