hey I think the method have changed since the last time. I am facing a problem when I try to create the migration file.

command used : docker exec -ti intelowl_uwsgi python3 manage.py dumpplugin AnalyzerConfig Netlas

error:

/usr/local/lib/python3.9/site-packages/dnstwist.py:64 No module named 'selenium'

/usr/local/lib/python3.9/site-packages/dnstwist.py:82 [Errno 2] No such file or directory: b'/usr/local/lib/python3.9/site-packages/GeoLite2-Country.mmdb'

/usr/local/lib/python3.9/site-packages/dnstwist.py:86 No module named 'GeoIP'

/usr/local/lib/python3.9/site-packages/dnstwist.py:114 No module named 'ssdeep'

XLMMacroDeobfuscator: pywin32 is not installed (only is required if you want to use MS Excel)

Traceback (most recent call last):

  File "/opt/deploy/intel_owl/manage.py", line 25, in <module>

    main()

  File "/opt/deploy/intel_owl/manage.py", line 21, in main

    execute_from_command_line(sys.argv)

  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line

    utility.execute()

  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute

    output = self.handle(*args, **options)

  File "/opt/deploy/intel_owl/api_app/management/commands/dumpplugin.py", line 227, in handle

    content = self._migration_file(obj, serializer_class, app)

  File "/opt/deploy/intel_owl/api_app/management/commands/dumpplugin.py", line 166, in _migration_file

    obj_data, param_data, values_data = self._get_serialization(

  File "/opt/deploy/intel_owl/api_app/management/commands/dumpplugin.py", line 47, in _get_serialization

    obj_data = serializer_class(obj).data

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 555, in data

    ret = super().data

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 253, in data

    self._data = self.to_representation(self.instance)

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 507, in to_representation

    for field in fields:

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 368, in _readable_fields

    for field in self.fields.values():

  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 356, in fields

    for key, value in self.get_fields().items():

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 1052, in get_fields

    field_names = self.get_field_names(declared_fields, info)

  File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 1112, in get_field_names

    raise TypeError(

TypeError: The `exclude` option must be a list or tuple. Got str.

Uhm, let me check it, I probably fucked up something during the Ingestor change to the script

Yeah, typo, solve with cc54e77.
Can you rebase from develop and let me know if you are able to create the migration?

thank you its working fine now!

Perfect!
As for the code, lgtm, the only change that I would like to have is some sort of valid mockup inside the analyzer (instead of a plain {}). A response taken from their api docs, instead of using your own api key is enough.

That and the new migration for the AnalyzerConfig. After these 2 things, we can merge the PR.

Codecov Report

Merging #1834 (7efe7d3) into develop (aa8820f) will increase coverage by 9.41%.
Report is 1280 commits behind head on develop.
The diff coverage is 76.13%.

@@             Coverage Diff             @@
##           develop    #1834      +/-   ##
===========================================
+ Coverage    66.75%   76.16%   +9.41%     
===========================================
  Files           95      381     +286     
  Lines         3706    12471    +8765     
  Branches       519     1314     +795     
===========================================
+ Hits          2474     9499    +7025     
- Misses         941     2430    +1489     
- Partials       291      542     +251     

... and 193 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c5b0902...7efe7d3. Read the comment docs.

Perfect! As for the code, lgtm, the only change that I would like to have is some sort of valid mockup inside the analyzer (instead of a plain {}). A response taken from their api docs, instead of using your own api key is enough.

That and the new migration for the AnalyzerConfig. After these 2 things, we can merge the PR.

I added the mock response. But when I am trying to commit the changes the black is running for the migration files.

error: cannot format api_app/ingestors_manager/migrations/0004_alter_ingestorreport_name.py: [Errno 13] Permission denied: 'api_app/ingestors_manager/migrations/0004_alter_ingestorreport_name.py'

error: cannot format api_app/analyzers_manager/migrations/0035_analyzer_config.py: [Errno 13] Permission denied: 'api_app/analyzers_manager/migrations/0035_analyzer_config.py'

Should I force commit or something?

That and the new migration for the AnalyzerConfig. After these 2 things, we can merge the PR.

sorry but what do you mean by new migration? I tried creating new migration but it says one already exists.

there are few more params we can add.

q

indices (optional)

start (optional)

fields

source_type

should I add them ?

Yeah, I should have added the thing about the permission.
When you are creating the migration inside the container, the file is created as root:root. If you go, in your host, inside api_app/analyzers_manager/migration you should be able to chown the file as your own, allowing the linter to correctly work.

That file is the migration file that it is missing in the PR: you should commit that too. And yes, it is correct the script it is not creating another migration because you already have made one for that AnalyzerConfig.

About the other params, as you wish. If you think that the user experience will benefit for having them, yes.

About the other params, as you wish. If you think that the user experience will benefit for having them, yes.

They aren't much specified in the API docs. I was not able to test them.
Is the mock response okay, I got it from the search in their app.

1 - The analyzer retrieve from the api response this structure result_data = result["items"][0]["data"]. In the mockup that you added, this dictionary structure is missing. You should change the mock of the raw request (meaning that you should have the at leastitems and data keys) or you should change the parsing of the request because right now they are not compatible.

will this work?
the response looks like this

2 - Unfortunately, you have pushed your own key because it was present in the database. You can change the value in the migration from your own key to None and revoke the key inside Netlas.

I was about to ask about API key being visible in the migration file. I changed it to null like others

The backend tests are failing. Is it due to my code?

Something wrong with the migration autogenerated, yey.
It is something on my end

Thanks @0ssigeno I thought the result would more match the mock_response. nvm. Do let me know if there is anything else we need to change.

LGTM, waiting let's see if @mlodic has some suggestions, otherwise I'm going to merge it

The PR is good, we are just missing some points in the checklist regarding the documentation.

• Usage file was updated.

• Advanced-Usage was updated (in case the plugin provides additional optional configuration).

okay, I think I messed up. I pulled and rebase these changes and the develop branch as well.

No big deal, i'm going to squash all together anyway.

The review is for the docs part