Ingestors by 0ssigeno · Pull Request #1809 · intelowlproject/IntelOwl

0ssigeno · 2023-07-31T14:50:15Z

No description provided.

Signed-off-by: 0ssigeno <[email protected]>

0ssigeno · 2023-07-31T14:50:29Z

TODO: docs

code-review-doctor

Looks good. Worth considering though. View full project report here.

api_app/ingestors_manager/models.py

0ssigeno · 2023-07-31T15:57:13Z

TODO celery beat scheduler

Signed-off-by: 0ssigeno <[email protected]>

codecov · 2023-08-01T07:58:23Z

Codecov Report

Merging #1809 (f1c81ce) into develop (aa8820f) will increase coverage by 9.45%.
Report is 1265 commits behind head on develop.
The diff coverage is 76.29%.

@@             Coverage Diff             @@
##           develop    #1809      +/-   ##
===========================================
+ Coverage    66.75%   76.21%   +9.45%     
===========================================
  Files           95      378     +283     
  Lines         3706    12380    +8674     
  Branches       519     1303     +784     
===========================================
+ Hits          2474     9435    +6961     
- Misses         941     2407    +1466     
- Partials       291      538     +247

Files Changed	Coverage Δ
...analyzers_manager/migrations/0004_datamigration.py	`91.30% <ø> (ø)`
api_app/decorators.py	`78.94% <ø> (ø)`
api_app/defaults.py	`100.00% <ø> (ø)`
api_app/documents.py	`100.00% <ø> (ø)`
api_app/fields.py	`57.14% <ø> (ø)`
api_app/filters.py	`72.97% <ø> (ø)`
api_app/forms.py	`37.50% <ø> (ø)`
api_app/helpers.py	`65.90% <ø> (+5.90%)`	⬆️
api_app/ingestors_manager/admin.py	`100.00% <ø> (ø)`
api_app/ingestors_manager/apps.py	`100.00% <ø> (ø)`
... and 231 more

... and 190 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc45f82...f1c81ce. Read the comment docs.

Signed-off-by: 0ssigeno <[email protected]>

code-review-doctor

Looks good. Worth considering though. View full project report here.

api_app/ingestors_manager/models.py

Signed-off-by: 0ssigeno <[email protected]>

intel_owl/tasks.py

docs/source/Usage.md

Signed-off-by: 0ssigeno <[email protected]>

mlodic

I report you this error too from the Admin section after clicking on "Crontab".

http://localhost/admin/django_celery_beat/crontabschedule/


Template error:
In template /usr/local/lib/python3.9/site-packages/django/contrib/admin/templates/admin/change_list.html, error at line 68
   
   58 :         {% block search %}{% search_form cl %}{% endblock %}
   59 :         {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %}
   60 : 
   61 :         <form id="changelist-form" method="post"{% if cl.formset and cl.formset.is_multipart %} enctype="multipart/form-data"{% endif %} novalidate>{% csrf_token %}
   62 :         {% if cl.formset %}
   63 :           <div>{{ cl.formset.management_form }}</div>
   64 :         {% endif %}
   65 : 
   66 :         {% block result_list %}
   67 :           {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %}
   68 :            {% result_list cl %} 
   69 :           {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %}
   70 :         {% endblock %}
   71 :         {% block pagination %}{% pagination cl %}{% endblock %}
   72 :         </form>
   73 :       </div>
   74 :       {% block filters %}
   75 :         {% if cl.has_filters %}
   76 :           <div id="changelist-filter">
   77 :             <h2>{% translate 'Filter' %}</h2>
   78 :             {% if cl.has_active_filters %}<h3 id="changelist-filter-clear">


Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/models/options.py", line 669, in get_field
    return self.fields_map[field_name]

During handling of the above exception ('human_readable'), another exception occurred:
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/utils.py", line 271, in lookup_field
    f = _get_non_gfk_field(opts, name)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/utils.py", line 302, in _get_non_gfk_field
    field = opts.get_field(name)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/options.py", line 671, in get_field
    raise FieldDoesNotExist(

During handling of the above exception (CrontabSchedule has no field named 'human_readable'), another exception occurred:
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 120, in get_full_description
    day_of_week_desc = self.get_day_of_week_description()
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 330, in get_day_of_week_description
    return self.get_segment_description(
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 503, in get_segment_description
    description_content += get_single_item_description(segment)
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 333, in <lambda>
    lambda s: get_day_name(s),
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 303, in get_day_name
    return ExpressionDescriptor.number_to_day(int(exp))

During handling of the above exception (invalid literal for int() with base 10: '[2'), another exception occurred:
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 220, in _get_response
    response = response.render()
  File "/usr/local/lib/python3.9/site-packages/django/template/response.py", line 114, in render
    self.content = self.rendered_content
  File "/usr/local/lib/python3.9/site-packages/django/template/response.py", line 92, in rendered_content
    return template.render(context, self._request)
  File "/usr/local/lib/python3.9/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 175, in render
    return self._render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/loader_tags.py", line 157, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/loader_tags.py", line 157, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/loader_tags.py", line 63, in render
    result = block.nodelist.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/loader_tags.py", line 63, in render
    result = block.nodelist.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/usr/local/lib/python3.9/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/templatetags/base.py", line 45, in render
    return super().render(context)
  File "/usr/local/lib/python3.9/site-packages/django/template/library.py", line 258, in render
    _dict = self.func(*resolved_args, **resolved_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/templatetags/admin_list.py", line 336, in result_list
    "results": list(results(cl)),
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/templatetags/admin_list.py", line 312, in results
    yield ResultList(None, items_for_result(cl, res, None))
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/templatetags/admin_list.py", line 303, in __init__
    super().__init__(*items)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/templatetags/admin_list.py", line 213, in items_for_result
    f, attr, value = lookup_field(field_name, result, cl.model_admin)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/admin/utils.py", line 282, in lookup_field
    attr = getattr(obj, name)
  File "/usr/local/lib/python3.9/site-packages/django_celery_beat/models.py", line 319, in human_readable
    human_readable = get_description('{} {} {} {} {}'.format(
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 652, in get_description
    return descripter.get_description(DescriptionTypeEnum.FULL)
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 104, in get_description
    return choices.get(description_type, self.get_seconds_description)()
  File "/usr/local/lib/python3.9/site-packages/cron_descriptor/ExpressionDescriptor.py", line 136, in get_full_description
    raise FormatException(description)

Exception Type: FormatException at /admin/django_celery_beat/crontabschedule/
Exception Value:

Signed-off-by: 0ssigeno <[email protected]>

Ingestor

943e0a8

Signed-off-by: 0ssigeno <[email protected]>

0ssigeno marked this pull request as draft July 31, 2023 14:50

code-review-doctor bot suggested changes Jul 31, 2023

View reviewed changes

api_app/ingestors_manager/models.py Show resolved Hide resolved

api_app/ingestors_manager/models.py Show resolved Hide resolved

0ssigeno added 2 commits July 31, 2023 17:57

Blake

25dcd30

Signed-off-by: 0ssigeno <[email protected]>

Fixes

acb7975

Signed-off-by: 0ssigeno <[email protected]>

0ssigeno added 9 commits August 3, 2023 08:41

Task for analyzers

8ee0e25

Signed-off-by: 0ssigeno <[email protected]>

More stuff

ee69457

Signed-off-by: 0ssigeno <[email protected]>

Blake

e4b2725

Signed-off-by: 0ssigeno <[email protected]>

Merge branch 'develop' into ingestor_v2

dbadfaa

Migrations

b202d15

Signed-off-by: 0ssigeno <[email protected]>

Fix tests

80eaff5

Signed-off-by: 0ssigeno <[email protected]>

Added docs

78d80a2

Signed-off-by: 0ssigeno <[email protected]>

Tests

dc96b2f

Signed-off-by: 0ssigeno <[email protected]>

Fix deepsource

2815980

Signed-off-by: 0ssigeno <[email protected]>

0ssigeno marked this pull request as ready for review August 4, 2023 08:38

0ssigeno requested a review from mlodic August 4, 2023 08:38

code-review-doctor bot suggested changes Aug 4, 2023

View reviewed changes

api_app/ingestors_manager/models.py Show resolved Hide resolved

api_app/ingestors_manager/models.py Show resolved Hide resolved

0ssigeno added 5 commits August 4, 2023 10:52

Typo :(

a8b432f

Signed-off-by: 0ssigeno <[email protected]>

Adding env variable

24baf74

Signed-off-by: 0ssigeno <[email protected]>

Added type

8a5c054

Signed-off-by: 0ssigeno <[email protected]>

Fixes

10c3db3

Signed-off-by: 0ssigeno <[email protected]>

Fix

676c953

Signed-off-by: 0ssigeno <[email protected]>

This was linked to issues Aug 4, 2023

New plugin type: Ingestors #1397

Closed

[Ingestor] Threatfox [requires #1397 completed] #1400

Closed

tweak to doc

1883903

mlodic reviewed Aug 4, 2023

View reviewed changes

intel_owl/tasks.py Outdated Show resolved Hide resolved

docs/source/Usage.md Outdated Show resolved Hide resolved

Fix

04fe768

Signed-off-by: 0ssigeno <[email protected]>

mlodic reviewed Aug 4, 2023

View reviewed changes

mlodic and others added 5 commits August 4, 2023 15:55

adjusted doc

6d59de1

Frontend + fixes

55750f9

Signed-off-by: 0ssigeno <[email protected]>

Prettier

749786c

Signed-off-by: 0ssigeno <[email protected]>

More docs

f1102fa

Signed-off-by: 0ssigeno <[email protected]>

Fix periodic task syntax

9d79fb9

Signed-off-by: 0ssigeno <[email protected]>

0ssigeno requested a review from mlodic August 7, 2023 07:06

mlodic approved these changes Aug 7, 2023

View reviewed changes

typo

f1c81ce

mlodic merged commit 0243f9f into develop Aug 7, 2023

0ssigeno deleted the ingestor_v2 branch September 6, 2023 15:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ingestors#1809

Ingestors#1809
mlodic merged 25 commits intodevelopfrom
ingestor_v2

0ssigeno commented Jul 31, 2023

Uh oh!

0ssigeno commented Jul 31, 2023

Uh oh!

code-review-doctor bot left a comment

Uh oh!

Uh oh!

Uh oh!

0ssigeno commented Jul 31, 2023

Uh oh!

codecov bot commented Aug 1, 2023 •

edited

Loading

Uh oh!

code-review-doctor bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlodic left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

0ssigeno commented Jul 31, 2023

Uh oh!

0ssigeno commented Jul 31, 2023

Uh oh!

code-review-doctor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

0ssigeno commented Jul 31, 2023

Uh oh!

codecov bot commented Aug 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

code-review-doctor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Aug 1, 2023 •

edited

Loading