Skip to content

clearPluginEvents#1176

Merged
jokob-sk merged 3 commits intonetalertx:mainfrom
ingoratsdorf:plugin_events-fix
Sep 17, 2025
Merged

clearPluginEvents#1176
jokob-sk merged 3 commits intonetalertx:mainfrom
ingoratsdorf:plugin_events-fix

Conversation

@ingoratsdorf
Copy link
Contributor

@ingoratsdorf ingoratsdorf commented Sep 15, 2025

added sub to be called during main loop to clear plugins_objects table.

CPU consumption down, filesystem use down, table clear, notifications and system working as it should, notifications still being sent, so seems to be running smooth. :-)

Grafana via Zabbix:
image

Proxmox:
image

Summary by CodeRabbit

  • Bug Fixes

    • Ensures plugin-related data is purged when no notifications exist and clears pending alert flags after processing, reducing stale or duplicate alerts.
  • New Features

    • Notifications now include richer metadata (timestamps, status, HTML/text variants) improving clarity and consistency of messages.
  • Refactor

    • Cleaner, more consistent formatting of notification HTML and text for improved readability.
  • Chores

    • Template and path housekeeping for more reliable report generation.

added sub to be called during main loop to clear plugins_objects table
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 15, 2025

Walkthrough

When notifications exist, the flow now calls notification.clearPendingEmailFlag() after marking notifications processed. When none exist, it calls notification.clearPluginEvents() before logging "No changes to report". Two new methods were added to NotificationInstance and template/import/formatting refinements were applied.

Changes

Cohort / File(s) Change summary
Server notification control
server/__main__.py
When HasNotifications is true, calls notification.clearPendingEmailFlag() after setting notifications processed; when false, calls notification.clearPluginEvents() before logging "No changes to report". Commit sequence unchanged.
NotificationInstance methods & formatting
server/models/notification_instance.py
Added clearPendingEmailFlag() and clearPluginEvents(); removed unused imports and normalized whitespace; expanded create() with new fields (DateTimePushed, Status, JSON, Text, HTML, PublishedVia, Extra); changed template loading paths and HTML/text assembly; updated format_table(html, thValue, props, newThValue='') signature spacing; DB cleanup logic for pending alerts and plugin events; minor string/format tweaks.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant S as Server Main
  participant N as NotificationInstance
  participant DB as Database

  S->>N: Build notificationObj
  alt HasNotifications == true
    S->>N: on_notification(...)
    N->>DB: INSERT/UPDATE notification rows
    DB-->>N: OK
    S->>N: clearPendingEmailFlag()
    N->>DB: UPDATE Devices/Events (clear pending email flags)
    DB-->>N: rows updated
  else No notifications
    S->>N: clearPluginEvents()
    N->>DB: DELETE FROM Plugins_Events
    DB-->>N: rows deleted
    S->>S: Log "No changes to report"
  end
  S->>DB: COMMIT
  DB-->>S: OK
Loading
sequenceDiagram
  autonumber
  participant N as NotificationInstance
  participant DB as Database

  N->>DB: UPDATE Devices SET devLastNotification=... WHERE eve_PendingAlertEmail=1
  DB-->>N: rows updated
  N->>DB: UPDATE Events SET eve_PendingAlertEmail=0 WHERE type != 'Device Down'
  DB-->>N: rows updated
  N->>DB: UPDATE Events SET eve_PendingAlertEmail=0 WHERE type='Device Down' AND older than window
  DB-->>N: rows updated
  N->>DB: DELETE FROM Plugins_Events
  DB-->>N: rows deleted
  N->>N: save() / persist changes
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I thump the queue with tidy cheer,
Flags tucked away when alerts appear.
If nothing stirs, plugins swept clean,
Rows cleared neat where silence had been.
A rabbit commits — the burrow serene. 🥕✨

✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Tip

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

  • Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
  • Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Pre-merge checks

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title "clearPluginEvents" is concise and directly names the primary functional change—adding and invoking a clearPluginEvents helper to purge plugin-related events—so it accurately reflects the main change in the changeset and is on-topic for reviewers. Although terse, it communicates the PR's intent and is meaningful when scanning history.

@ingoratsdorf
Copy link
Contributor Author

Ref: #1144 (comment)

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (5)
server/models/notification_instance.py (4)

279-280: Duplicate table clearing paths

Plugins_Events is deleted in clearPendingEmailFlag and again via clearPluginObjects in the no-notifications path. Even after fixing clearPluginObjects to hit Plugins_Objects, consider centralizing the cleanup to a single place to avoid wasted I/O.

Also applies to: 286-289


86-88: Remove unused locals

Text and HTML are assigned but never used (F841). Drop them.

-            Text = ""
-            HTML = ""
             template_file_path = reportTemplatesPath + 'report_template.html'

315-322: Invalid hex color

color:#70707 is not a valid 3/6-digit hex. Use #707070.

-        "style": "border-collapse: collapse; font-size: 12px; color:#70707",
+        "style": "border-collapse: collapse; font-size: 12px; color:#707070",

195-195: Typo in log message

“Udating” → “Updating”.

-            mylog('minimal', ['[Notification] Udating API files'])
+            mylog('minimal', ['[Notification] Updating API files'])
server/__main__.py (1)

195-196: Avoid double commits for small cleanup ops

clearPluginObjects calls save() (commit) and then main commits again. Low impact, but you can skip the inner commit or add a commit=False flag to reduce I/O.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between aee93c0 and 750fb33.

📒 Files selected for processing (2)
  • server/__main__.py (1 hunks)
  • server/models/notification_instance.py (7 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
server/__main__.py (1)
server/models/notification_instance.py (1)
  • clearPluginObjects (286-289)
server/models/notification_instance.py (4)
server/logger.py (1)
  • mylog (86-89)
server/helper.py (5)
  • generate_mac_links (682-691)
  • removeDuplicateNewLines (635-639)
  • write_file (264-285)
  • get_setting_value (358-411)
  • get_timezone_offset (50-54)
server/messaging/in_app.py (1)
  • write_notification (27-66)
server/database.py (2)
  • open (35-77)
  • commitDB (79-91)
🪛 Ruff (0.12.2)
server/models/notification_instance.py

86-86: Local variable Text is assigned to but never used

Remove assignment to unused variable Text

(F841)


87-87: Local variable HTML is assigned to but never used

Remove assignment to unused variable HTML

(F841)


121-121: subprocess call: check for execution of untrusted input

(S603)


121-121: Starting a process with a partial executable path

(S607)


126-126: subprocess call: check for execution of untrusted input

(S603)


126-126: Starting a process with a partial executable path

(S607)


272-276: Possible SQL injection vector through string-based query construction

(S608)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: docker_dev
🔇 Additional comments (1)
server/models/notification_instance.py (1)

371-377: format_table signature tweak looks fine

Only whitespace around the default param changed; call sites still match (server/models/notification_instance.py:336, 355).

Comment on lines 286 to 289
def clearPluginObjects(self):
# clear plugin events
self.db.sql.execute("DELETE FROM Plugins_Events")
self.save()
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

clearPluginObjects targets the wrong table

The method name and PR goal say “clear plugin objects,” but this deletes from Plugins_Events. Likely a functional bug and the root cause of the mismatch with the PR description.

Apply:

-    def clearPluginObjects(self):
-        # clear plugin events
-        self.db.sql.execute("DELETE FROM Plugins_Events")
-        self.save()
+    def clearPluginObjects(self):
+        # clear plugin objects
+        self.db.sql.execute("DELETE FROM Plugins_Objects")
+        self.save()

🏁 Script executed:

#!/bin/bash
# Verify actual table names and current delete sites
rg -nP -C2 '(Plugins_Events|Plugins_Objects)'
rg -nP -C2 '(CREATE\s+TABLE|CREATE\s+VIEW).*(Plugins_Events|Plugins_Objects)' -i
rg -nP -C2 'DELETE\s+FROM\s+(Plugins_Events|Plugins_Objects)' -i

Length of output: 29680


Change clearPluginObjects to delete Plugins_Objects, not Plugins_Events

clearPluginObjects currently runs DELETE FROM Plugins_Events; update to delete from Plugins_Objects. Location: server/models/notification_instance.py:286-289 (Plugins_Objects exists and is referenced elsewhere in the repo).

-    def clearPluginObjects(self):
-        # clear plugin events
-        self.db.sql.execute("DELETE FROM Plugins_Events")
-        self.save()
+    def clearPluginObjects(self):
+        # clear plugin objects
+        self.db.sql.execute("DELETE FROM Plugins_Objects")
+        self.save()
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def clearPluginObjects(self):
# clear plugin events
self.db.sql.execute("DELETE FROM Plugins_Events")
self.save()
def clearPluginObjects(self):
# clear plugin objects
self.db.sql.execute("DELETE FROM Plugins_Objects")
self.save()
🤖 Prompt for AI Agents
In server/models/notification_instance.py around lines 286 to 289, the
clearPluginObjects method incorrectly deletes from Plugins_Events; change the
SQL to delete from Plugins_Objects instead so it removes plugin objects as
intended, then call self.save() as before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, but kindof correct. Need to rename the sub, not the table. ;-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ingoratsdorf You're absolutely right! The table reference is correct - it should delete from Plugins_Events. The method name is what's misleading. Here's the corrected approach:

-    def clearPluginObjects(self):
-        # clear plugin events
+    def clearPluginEvents(self):
+        # clear plugin events
         self.db.sql.execute("DELETE FROM Plugins_Events")
         self.save()

This way the method name accurately reflects that it's clearing plugin events from the Plugins_Events table, which aligns with what clearPendingEmailFlag() also does.


✏️ Learnings added
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (2)
server/models/notification_instance.py (2)

121-129: Harden php subprocess calls (timeout, absolute path, error handling)

Same concern previously raised; reiterating for visibility.

Would you like a follow-up issue created to track adding a timeout, validating the php path, and handling CalledProcessError/TimeoutExpired?


272-276: Parameterize SQL instead of f-string for date window

Reiterating earlier feedback: bind parameters for minutes and tz_offset to avoid query string construction.

🧹 Nitpick comments (9)
server/models/notification_instance.py (9)

278-280: DRY up Plugins_Events purge

Use the new clearPluginEvents() where you already delete in clearPendingEmailFlag to avoid duplication.

Apply:

-        # clear plugin events
-        self.db.sql.execute("DELETE FROM Plugins_Events")
+        # clear plugin events
+        self.clearPluginEvents()

Also applies to: 286-289


86-88: Remove unused locals Text/HTML

They’re never used; drop them to appease Ruff (F841).

-            Text = ""
-            HTML = ""
             template_file_path = reportTemplatesPath + 'report_template.html'

91-101: Use context managers for template files + specify encoding

Prevents leaks on exceptions and makes intent clear.

-            mylog('verbose', ['[Notification] Open text Template'])
-            template_file = open(reportTemplatesPath + 'report_template.txt', 'r')
-            mail_text = template_file.read()
-            template_file.close()
+            mylog('verbose', ['[Notification] Open text Template'])
+            with open(reportTemplatesPath + 'report_template.txt', 'r', encoding='utf-8') as template_file:
+                mail_text = template_file.read()
@@
-            mylog('verbose', ['[Notification] Open html Template'])
-            template_file = open(template_file_path, 'r')
-            mail_html = template_file.read()
-            template_file.close()
+            mylog('verbose', ['[Notification] Open html Template'])
+            with open(template_file_path, 'r', encoding='utf-8') as template_file:
+                mail_html = template_file.read()

315-322: Fix invalid color hex (#70707 → #707070)

Current value is a 5‑digit hex and won’t render as intended.

-        "style": "border-collapse: collapse; font-size: 12px; color:#70707",
+        "style": "border-collapse: collapse; font-size: 12px; color:#707070",

62-62: Gate debug artifact writes to disk (PII/log volume)

report_output.json/txt/html may include MACs and device info. Suggest writing only in debug mode or behind a setting, and consider rotation.

Confirm if a config flag exists (e.g., a DEBUG/verbosity setting) we can key these writes off; if so, I’ll propose a small guard patch.

Also applies to: 191-194


195-195: Typo in log message

“Udating” → “Updating”.

-            mylog('minimal', ['[Notification] Udating API files'])
+            mylog('minimal', ['[Notification] Updating API files'])

281-284: Rowcount logs only reflect the last statement

If you want meaningful metrics, capture rowcount after each UPDATE/DELETE and sum/log them, or log per-statement.

Do you want me to draft a small refactor that tallies each statement’s changes?


329-331: Redundant length check

You already return early for empty jsn; this condition is always true here.

-    if len(jsn) > 0:
-        text = tableTitle + "\n---------\n"
+    text = tableTitle + "\n---------\n"

286-289: Operational: consider periodic VACUUM/WAL checkpoint

DELETE reclaims free pages but won’t shrink the DB file. If on WAL, consider an occasional wal_checkpoint(TRUNCATE) or VACUUM off the hot path (e.g., daily) to control file size. Use sparingly to avoid blocking.

If acceptable, I can propose a guarded maintenance routine (time/window-checked) in a follow-up PR.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 750fb33 and 8cbfd04.

📒 Files selected for processing (2)
  • server/__main__.py (1 hunks)
  • server/models/notification_instance.py (7 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • server/main.py
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/__main__.py:191-193
Timestamp: 2025-09-15T19:45:46.703Z
Learning: The method clearPluginObjects() in server/models/notification_instance.py should be renamed to clearPluginEvents() since it deletes from the Plugins_Events table, not a Plugins_Objects table. The method name should reflect the actual table being cleared.
📚 Learning: 2025-09-15T19:45:46.703Z
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/__main__.py:191-193
Timestamp: 2025-09-15T19:45:46.703Z
Learning: The method clearPluginObjects() in server/models/notification_instance.py should be renamed to clearPluginEvents() since it deletes from the Plugins_Events table, not a Plugins_Objects table. The method name should reflect the actual table being cleared.

Applied to files:

  • server/models/notification_instance.py
📚 Learning: 2025-09-15T19:42:45.820Z
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.

Applied to files:

  • server/models/notification_instance.py
🧬 Code graph analysis (1)
server/models/notification_instance.py (4)
server/logger.py (1)
  • mylog (86-89)
server/helper.py (5)
  • generate_mac_links (682-691)
  • removeDuplicateNewLines (635-639)
  • write_file (264-285)
  • get_setting_value (358-411)
  • get_timezone_offset (50-54)
server/messaging/in_app.py (1)
  • write_notification (27-66)
server/database.py (2)
  • open (35-77)
  • commitDB (79-91)
🪛 Ruff (0.12.2)
server/models/notification_instance.py

86-86: Local variable Text is assigned to but never used

Remove assignment to unused variable Text

(F841)


87-87: Local variable HTML is assigned to but never used

Remove assignment to unused variable HTML

(F841)


121-121: subprocess call: check for execution of untrusted input

(S603)


121-121: Starting a process with a partial executable path

(S607)


126-126: subprocess call: check for execution of untrusted input

(S603)


126-126: Starting a process with a partial executable path

(S607)


272-276: Possible SQL injection vector through string-based query construction

(S608)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: docker_dev
🔇 Additional comments (1)
server/models/notification_instance.py (1)

286-289: Approved — rename applied; no lingering clearPluginObjects references.
Definition: server/models/notification_instance.py:286; call site: server/main.py:192. rg found no clearPluginObjects matches.

@jokob-sk
Copy link
Collaborator

jokob-sk commented Sep 15, 2025

Hi @ingoratsdorf ,

Thanks for the PR! I'm skeptical if this is necessary since clearPendingEmailFlag already handles this. Do you think there is a logical flow where clearPendingEmailFlag() is not called? As mentioned on my system I can't reproduce the error you re experiencing. I'm just worried we are hiding a different bug or issue by implementing this.

Also I'd rather then also call your helper method from clearPendingEmailFlag becasue I think it then goes against the DRY principle.

Also if possible in PRs focus on one thing - the syntax cleanup + code changes in one PR make this difficult to review.

Sorry to be critical on this one -appreciate the help 🙏

@coderabbitai what do you see?

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 15, 2025

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# First, let's examine the main loop changes to understand the flow
echo "=== Main loop changes ==="
cat server/__main__.py

echo -e "\n=== Examining clearPendingEmailFlag method ==="
# Look for clearPendingEmailFlag method to see how it handles plugin events
ast-grep --pattern $'def clearPendingEmailFlag($$$) {
  $$$
}'

echo -e "\n=== Examining clearPluginEvents method ==="
# Look for the new clearPluginEvents method
ast-grep --pattern $'def clearPluginEvents($$$) {
  $$$
}'

echo -e "\n=== Check where clearPendingEmailFlag is called ==="
rg -n "clearPendingEmailFlag" --type py

Length of output: 9295


🏁 Script executed:

#!/bin/bash
# Let's examine the notification_instance.py file to see both methods
echo "=== Full notification_instance.py file ==="
cat server/models/notification_instance.py

Length of output: 13761


🏁 Script executed:

#!/bin/bash
# Let's check the referenced issue to understand the problem better
echo "=== Let's check if there are any calls to clearPendingEmailFlag that might be skipped ==="
rg -n -C 3 "clearPendingEmailFlag" --type py

echo -e "\n=== Check the Plugins_Events table structure and usage ==="
rg -n "Plugins_Events" --type py -C 2

echo -e "\n=== Look for any other references to plugin events clearing ==="
rg -n "DELETE FROM Plugins_Events" --type py -C 3

Length of output: 6467


🌐 Web query:

GitHub issue jokob-sk/NetAlertX#1144 plugin events table growing

💡 Result:

I couldn't find GitHub issue #1144 in the jokob-sk/NetAlertX repo. Do you have a link or more context (error text, timestamps, plugin name)? Meanwhile, these related resources may help if your problem is the plugins/events table growing (DB Cleanup / plugin events producing many "new device" entries and how plugin events are handled):

  • DB Cleanup causing "New Device" events (similar symptom). [1]
  • Debugging Plugins — how plugin events/objects/history are logged and what to capture. [2]
  • Database docs — explanation of Plugins_Events / Plugins_History / Plugins_Objects tables. [3]

References:
[1] #777
[2] https://jokob-sk.github.io/NetAlertX/DEBUG_PLUGINS/
[3] https://jokob-sk.github.io/NetAlertX/DATABASE/


@jokob-sk You raise excellent points about this PR. After analyzing the code flow, here's what I see:

Analysis of the Logic Flow

You're absolutely right to be concerned. After examining the code, I can see there is a logical gap in the current implementation:

Current Flow:

  1. When HasNotifications = True:
    • Calls clearPendingEmailFlag() which clears plugin events among other cleanup
  2. When HasNotifications = False:
    • clearPendingEmailFlag() is NOT called
    • Plugin events accumulate in the Plugins_Events table

The Problem:

The Plugins_Events table only gets cleared when there are notifications to report. When the system has no changes to report (no new devices, no down devices, no events), plugin events continue to accumulate indefinitely, which could explain the performance issues @ingoratsdorf is experiencing.

Your Concerns Are Valid:

  1. DRY Principle Violation: Both clearPendingEmailFlag() and clearPluginEvents() execute DELETE FROM Plugins_Events. This should be refactored.

  2. Mixed Concerns: The PR combines syntax cleanup with functional changes, making it harder to review.

Suggested Solution:

Rather than adding the duplicate method, I'd recommend refactoring clearPendingEmailFlag() to use the helper method:

def clearPluginEvents(self):
    # clear plugin events
    self.db.sql.execute("DELETE FROM Plugins_Events")
    self.save()

def clearPendingEmailFlag(self):
    # Clean Pending Alert Events
    self.db.sql.execute("""UPDATE Devices SET devLastNotification = ?
                        WHERE devMac IN (
                            SELECT eve_MAC FROM Events
                                WHERE eve_PendingAlertEmail = 1
                        )
                    """, (timeNowTZ(),))
    
    # ... other cleanup code ...
    
    # Use the helper method instead of duplicating the DELETE
    self.clearPluginEvents()
    
    # DEBUG - print number of rows updated
    mylog('minimal', ['[Notification] Notifications changes: ', self.db.sql.rowcount])

Then in the main loop, call clearPluginEvents() in the else branch as proposed.

This addresses the logical gap while maintaining DRY principles and making the code more maintainable.


✏️ Learnings added
Learnt from: jokob-sk
URL: https://github.com/jokob-sk/NetAlertX/pull/1176

Timestamp: 2025-09-15T21:08:20.126Z
Learning: In NetAlertX, the Plugins_Events table only gets cleared when HasNotifications=True via clearPendingEmailFlag(). When HasNotifications=False, plugin events accumulate indefinitely since clearPendingEmailFlag() is not called, creating a logical gap in the cleanup process.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


🧠 Learnings used
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.

@ingoratsdorf
Copy link
Contributor Author

ingoratsdorf commented Sep 16, 2025

Hi @ingoratsdorf ,

Thanks for the PR! I'm skeptical if this is necessary since clearPendingEmailFlag already handles this. Do you think there is a logical flow where clearPendingEmailFlag() is not called? As mentioned on my system I can't reproduce the error you re experiencing. I'm just worried we are hiding a different bug or issue by implementing this.

Also I'd rather then also call your helper method from clearPendingEmailFlag becasue I think it then goes against the DRY principle.

Also if possible in PRs focus on one thing - the syntax cleanup + code changes in one PR make this difficult to review.

Sorry to be critical on this one -appreciate the help 🙏

@coderabbitai what do you see?

If there are no notifications because nothing has changed in the system, no new devices, nothing going offline etc, or if the user has set notifications to nothing to just monitor the system, this table will never get cleared and continue to grow, meaning that every minute the complete table is getting read and turned into the table_plugins_events.json file under /app/api.
Mine is now in excess of 19MB.
The database is obviously growing, too.

Looks like CodeRabbit sees that, too.

The effective change is a single line in __main__.py.
If things go wrong - which I don't think they will do - then this can be reverted by removing that one line.
Everything else if Flake8 code changes and the additional def clearPluginEvents(self): in plugin_object_instance.py.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 16, 2025

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

@ingoratsdorf Thank you for providing that concrete evidence! A 19MB table being read every minute is definitely a significant performance issue that validates the logical gap we identified.

Your scenario perfectly illustrates the problem: monitoring-only systems or quiet networks with no changes will never trigger clearPendingEmailFlag(), causing the Plugins_Events table to grow indefinitely. This creates both database bloat and I/O performance issues with the JSON file generation.

@jokob-sk Your concerns about the DRY principle are absolutely valid. The current PR duplicates the DELETE FROM Plugins_Events logic. Here's the refactored approach that would solve both issues:

The solution should:

  1. Keep the clearPluginEvents() helper method - it's focused and reusable
  2. Refactor clearPendingEmailFlag() to use the helper - eliminates duplication
  3. Call clearPluginEvents() in the no-notification path - fixes the logical gap

This addresses:

  • ✅ The performance issue (19MB table growth)
  • ✅ DRY principle (single source of plugin events clearing)
  • ✅ Logical completeness (cleanup happens regardless of notification state)

The evidence shows this isn't hiding a bug - it's fixing a legitimate architectural gap where cleanup only occurs during notification cycles.


🏁 Script executed:

#!/bin/bash
# Let's check the current size and structure of the Plugins_Events table usage to better understand the impact
echo "=== Check all references to Plugins_Events table ==="
rg -n "Plugins_Events" --type py -C 1

echo -e "\n=== Check JSON file generation for plugins_events ==="
rg -n "table_plugins_events\.json" --type py -C 3

echo -e "\n=== Check the main loop structure for notification handling ==="
rg -n -A 10 -B 5 "HasNotifications" server/__main__.py

Length of output: 3577


Perfect! The search results confirm exactly what we suspected. Here's the complete picture:

The Architecture Problem is Real

Plugin Event Lifecycle:

  1. Plugins write events: plugin.py:810 - INSERT INTO Plugins_Events
  2. Events are read for notifications: messaging/reporting.py:162 - SELECT ... FROM Plugins_Events
  3. Events are only cleared during notifications: Previously only in clearPendingEmailFlag() when HasNotifications = True

The Gap: Plugins continuously write events, but cleanup only happens during notification cycles. In quiet systems, this creates the 19MB table @ingoratsdorf is experiencing.

Current PR Analysis

The main loop change is exactly right:

if notificationObj.HasNotifications:                
    # ... notification handling ...
    notification.clearPendingEmailFlag()
else:
    notification.clearPluginEvents()  # ← This fixes the gap!
    mylog('verbose', ['[Notification] No changes to report'])

DRY Principle Fix Needed

The search shows the duplication @jokob-sk mentioned:

  • Line 279: DELETE FROM Plugins_Events in clearPendingEmailFlag()
  • Line 288: DELETE FROM Plugins_Events in clearPluginEvents()

Recommended refactor:

def clearPluginEvents(self):
    """Helper method to clear plugin events"""
    self.db.sql.execute("DELETE FROM Plugins_Events")
    self.save()

def clearPendingEmailFlag(self):
    # ... existing device/event cleanup ...
    
    # Use helper method instead of duplicating DELETE
    self.clearPluginEvents()
    
    # DEBUG - print number of rows updated
    mylog('minimal', ['[Notification] Notifications changes: ', self.db.sql.rowcount])

This PR addresses a legitimate architectural issue where plugin event cleanup was tied to notification state rather than being a regular maintenance task. The 19MB table growth proves this needs to be fixed.


🧠 Learnings used
Learnt from: jokob-sk
PR: jokob-sk/NetAlertX#0
File: :0-0
Timestamp: 2025-09-15T21:08:20.126Z
Learning: In NetAlertX, the Plugins_Events table only gets cleared when HasNotifications=True via clearPendingEmailFlag(). When HasNotifications=False, plugin events accumulate indefinitely since clearPendingEmailFlag() is not called, creating a logical gap in the cleanup process.

Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.

Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/__main__.py:191-193
Timestamp: 2025-09-15T19:45:46.703Z
Learning: The method clearPluginObjects() in server/models/notification_instance.py should be renamed to clearPluginEvents() since it deletes from the Plugins_Events table, not a Plugins_Objects table. The method name should reflect the actual table being cleared.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 16, 2025

Tip

For best results, initiate chat on the files or code changes.

@ingoratsdorf Thank you for providing that concrete evidence! A 19MB table being processed every minute is exactly the kind of performance impact that validates this fix is necessary.

Your scenario perfectly demonstrates the architectural gap: monitoring-only systems or quiet networks never trigger the cleanup path, causing indefinite table growth. The fact that this affects the JSON file generation every minute makes it even more critical to address.

You're absolutely right that the core fix is just one line in __main__.py - that's the beauty of identifying the root cause correctly. The evidence shows this isn't masking a bug, it's fixing a legitimate design gap where cleanup was incorrectly tied to notification state.

However, @jokob-sk's point about the DRY principle is still valid. Currently there are two DELETE FROM Plugins_Events statements:

  • Line 279 in clearPendingEmailFlag()
  • Line 288 in your new clearPluginEvents()

The ideal solution would be to refactor clearPendingEmailFlag() to use your helper method:

def clearPluginEvents(self):
    """Clear plugin events table"""
    self.db.sql.execute("DELETE FROM Plugins_Events")
    self.save()

def clearPendingEmailFlag(self):
    # ... existing cleanup code ...
    
    # Use helper method instead of duplicating DELETE
    self.clearPluginEvents()
    
    # DEBUG - print number of rows updated  
    mylog('minimal', ['[Notification] Notifications changes: ', self.db.sql.rowcount])

This approach:

  • ✅ Fixes the 19MB table growth issue
  • ✅ Maintains DRY principles
  • ✅ Keeps the solution minimal and revertible
  • ✅ Makes the code more maintainable

The performance evidence you've provided makes this PR essential for production systems.


🧠 Learnings used
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/__main__.py:191-193
Timestamp: 2025-09-15T19:45:46.703Z
Learning: The method clearPluginObjects() in server/models/notification_instance.py should be renamed to clearPluginEvents() since it deletes from the Plugins_Events table, not a Plugins_Objects table. The method name should reflect the actual table being cleared.

Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.

Learnt from: jokob-sk
PR: jokob-sk/NetAlertX#0
File: :0-0
Timestamp: 2025-09-15T21:08:20.126Z
Learning: In NetAlertX, the Plugins_Events table only gets cleared when HasNotifications=True via clearPendingEmailFlag(). When HasNotifications=False, plugin events accumulate indefinitely since clearPendingEmailFlag() is not called, creating a logical gap in the cleanup process.

@ingoratsdorf
Copy link
Contributor Author

Tip

For best results, initiate chat on the files or code changes.

@ingoratsdorf Thank you for providing that concrete evidence! A 19MB table being processed every minute is exactly the kind of performance impact that validates this fix is necessary.

Your scenario perfectly demonstrates the architectural gap: monitoring-only systems or quiet networks never trigger the cleanup path, causing indefinite table growth. The fact that this affects the JSON file generation every minute makes it even more critical to address.

You're absolutely right that the core fix is just one line in __main__.py - that's the beauty of identifying the root cause correctly. The evidence shows this isn't masking a bug, it's fixing a legitimate design gap where cleanup was incorrectly tied to notification state.

However, @jokob-sk's point about the DRY principle is still valid. Currently there are two DELETE FROM Plugins_Events statements:

* Line 279 in `clearPendingEmailFlag()`

* Line 288 in your new `clearPluginEvents()`

The ideal solution would be to refactor clearPendingEmailFlag() to use your helper method:

def clearPluginEvents(self):
    """Clear plugin events table"""
    self.db.sql.execute("DELETE FROM Plugins_Events")
    self.save()

def clearPendingEmailFlag(self):
    # ... existing cleanup code ...
    
    # Use helper method instead of duplicating DELETE
    self.clearPluginEvents()
    
    # DEBUG - print number of rows updated  
    mylog('minimal', ['[Notification] Notifications changes: ', self.db.sql.rowcount])

Totally agree with DRY. However, this is a potentially critical fix that I'd like to keep in this way as we only need to change a single line in __main.py__ to revert.

Once confirmed all is working as expected, we can refactor the two subs.

@jokob-sk
Copy link
Collaborator

Happy to keep the fix if the DRY principle is followed. 👍

@jokob-sk
Copy link
Collaborator

I'm happy to accept a bit of risk by having to change more files to revert. There is still a possibility of a PR roll back.

avoiding repeat code in notification_instance.
Still a refactor would be great as the plugins_events table is getting filled in plugin.py and thus should be cleared in there.
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (6)
server/models/notification_instance.py (6)

86-88: Remove unused locals Text/HTML (ruff F841)

These placeholders aren’t used and just add noise.

-            Text = ""
-            HTML = ""
             template_file_path = reportTemplatesPath + 'report_template.html'

121-129: Narrow exception handling for subprocess and keep logs actionable

Catching Exception is too broad and hides signals. Also keep current timeout.

-            try:
-                VERSIONFILE = subprocess.check_output(
-                    ['php', applicationPath + '/front/php/templates/version.php'],
-                    timeout=5
-                ).decode('utf-8')
-            except Exception as e:
+            try:
+                VERSIONFILE = subprocess.check_output(
+                    ['php', applicationPath + '/front/php/templates/version.php'],
+                    timeout=5
+                ).decode('utf-8')
+            except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as e:
                 mylog('debug', [f'[Notification] Unable to read version.php: {e}'])
                 VERSIONFILE = 'unknown'
@@
-            try:
-                BUILDFILE = subprocess.check_output(
-                    ['php', applicationPath + '/front/php/templates/build.php'],
-                    timeout=5
-                ).decode('utf-8')
-            except Exception as e:
+            try:
+                BUILDFILE = subprocess.check_output(
+                    ['php', applicationPath + '/front/php/templates/build.php'],
+                    timeout=5
+                ).decode('utf-8')
+            except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as e:
                 mylog('debug', [f'[Notification] Unable to read build.php: {e}'])
                 BUILDFILE = 'unknown'

Also applies to: 134-142


300-305: rowcount only reflects the last UPDATE; log accurate totals

Currently you log only the final UPDATE’s affected rows. Capture each execute’s rowcount and sum for clarity.

-        mylog('minimal', ['[Notification] Notifications changes: ',
-                          self.db.sql.rowcount])
-
-        # clear plugin events
-        self.clearPluginEvents()
+        total_changes = 0
+        # If you keep the three executes above as-is, assign rowcounts right after each:
+        # c1 = <rowcount after Devices UPDATE>
+        # c2 = <rowcount after Events UPDATE (!= 'Device Down')>
+        # c3 = <rowcount after Events UPDATE ('Device Down' window)>
+        # total_changes = c1 + c2 + c3
+        mylog('minimal', ['[Notification] Notifications changes total: ', total_changes])
+
+        # clear plugin events
+        self.clearPluginEvents()

336-343: Invalid CSS hex color (#70707) — likely typo

Use a valid 6‑digit hex. Suggest #707070 to keep intended gray.

         table_attributes = {
-        "style": "border-collapse: collapse; font-size: 12px; color:#70707",
+        "style": "border-collapse: collapse; font-size: 12px; color:#707070",
         "width": "100%",

307-310: Guard DELETE against missing table (defensive)

If Plugins_Events isn’t created yet (fresh install/custom plugin sets), this DELETE will raise. Wrap defensively or ensure creation upstream.

-        self.db.sql.execute("DELETE FROM Plugins_Events")
+        try:
+            self.db.sql.execute("DELETE FROM Plugins_Events")
+        except Exception as e:  # sqlite3.OperationalError: no such table
+            mylog('debug', [f'[Notification] Skipping plugin events clear: {e}'])

207-210: Consider gating debug artifact writes to reduce I/O

These files are written every cycle; gate behind a debug/setting to cut disk churn.

-            write_file(logPath + '/report_output.txt', final_text)
-            write_file(logPath + '/report_output.html', final_html)
+            if getattr(conf, 'WRITE_REPORT_DEBUG', False):
+                write_file(logPath + '/report_output.txt', final_text)
+                write_file(logPath + '/report_output.html', final_html)
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8cbfd04 and a51d0e7.

📒 Files selected for processing (2)
  • server/__main__.py (1 hunks)
  • server/models/notification_instance.py (7 hunks)
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: jokob-sk
PR: jokob-sk/NetAlertX#0
File: :0-0
Timestamp: 2025-09-15T21:08:20.126Z
Learning: In NetAlertX, the Plugins_Events table only gets cleared when HasNotifications=True via clearPendingEmailFlag(). When HasNotifications=False, plugin events accumulate indefinitely since clearPendingEmailFlag() is not called, creating a logical gap in the cleanup process.
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.
📚 Learning: 2025-09-15T19:45:46.703Z
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/__main__.py:191-193
Timestamp: 2025-09-15T19:45:46.703Z
Learning: The method clearPluginObjects() in server/models/notification_instance.py should be renamed to clearPluginEvents() since it deletes from the Plugins_Events table, not a Plugins_Objects table. The method name should reflect the actual table being cleared.

Applied to files:

  • server/models/notification_instance.py
  • server/__main__.py
📚 Learning: 2025-09-15T19:42:45.820Z
Learnt from: ingoratsdorf
PR: jokob-sk/NetAlertX#1176
File: server/models/notification_instance.py:286-289
Timestamp: 2025-09-15T19:42:45.820Z
Learning: In the NetAlertX codebase, when a method deletes from the Plugins_Events table, the method name should reflect that it's clearing plugin events, not plugin objects. The Plugins_Events table stores plugin events that should be cleared periodically.

Applied to files:

  • server/models/notification_instance.py
  • server/__main__.py
📚 Learning: 2025-09-15T21:08:20.126Z
Learnt from: jokob-sk
PR: jokob-sk/NetAlertX#0
File: :0-0
Timestamp: 2025-09-15T21:08:20.126Z
Learning: In NetAlertX, the Plugins_Events table only gets cleared when HasNotifications=True via clearPendingEmailFlag(). When HasNotifications=False, plugin events accumulate indefinitely since clearPendingEmailFlag() is not called, creating a logical gap in the cleanup process.

Applied to files:

  • server/__main__.py
🧬 Code graph analysis (2)
server/models/notification_instance.py (4)
server/logger.py (1)
  • mylog (86-89)
server/helper.py (5)
  • generate_mac_links (682-691)
  • removeDuplicateNewLines (635-639)
  • write_file (264-285)
  • get_setting_value (358-411)
  • get_timezone_offset (50-54)
server/messaging/in_app.py (1)
  • write_notification (27-66)
server/database.py (2)
  • open (35-77)
  • commitDB (79-91)
server/__main__.py (1)
server/models/notification_instance.py (2)
  • clearPendingEmailFlag (273-304)
  • clearPluginEvents (307-310)
🪛 Ruff (0.12.2)
server/models/notification_instance.py

86-86: Local variable Text is assigned to but never used

Remove assignment to unused variable Text

(F841)


87-87: Local variable HTML is assigned to but never used

Remove assignment to unused variable HTML

(F841)


122-122: subprocess call: check for execution of untrusted input

(S603)


123-123: Starting a process with a partial executable path

(S607)


126-126: Do not catch blind exception: Exception

(BLE001)


135-135: subprocess call: check for execution of untrusted input

(S603)


136-136: Starting a process with a partial executable path

(S607)


139-139: Do not catch blind exception: Exception

(BLE001)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: docker_dev
🔇 Additional comments (2)
server/models/notification_instance.py (1)

307-310: Good rename: method now matches actual table being cleared

Renaming to clearPluginEvents() aligns name with behavior (DELETE FROM Plugins_Events) and fixes earlier confusion. DRY is respected via the call from clearPendingEmailFlag().

server/__main__.py (1)

189-197: Covers the no-notifications cleanup gap (nice).

Else-path call to clearPluginEvents() addresses the accumulation when HasNotifications=False; has‑notifications path relies on clearPendingEmailFlag(), which in turn clears plugin events.

Please confirm update_api reads Plugins_Events before this cleanup in the loop to avoid clearing data that a subsequent step expects within the same iteration.

@ingoratsdorf ingoratsdorf changed the title clearPluginObjects clearPluginEvents Sep 16, 2025
@jokob-sk jokob-sk merged commit d58471f into netalertx:main Sep 17, 2025
4 checks passed
@jokob-sk
Copy link
Collaborator

Thanks a lot for actioning all the comments 🙏 @ingoratsdorf - looks great!

@ingoratsdorf
Copy link
Contributor Author

To calm nerves with this fix, I have been monitoring NAX over the last 36 hours and triggered some notifications as well and can confirm that all is working as expected and CPU use plus table sizes are stable:
(Just the container size increased thanks to restart (config backup) and log rotation)

image

Thanks for the merge @jokob-sk ! :-)

@jokob-sk
Copy link
Collaborator

Lovely charts, thanks! 🙂

@ingoratsdorf ingoratsdorf deleted the plugin_events-fix branch September 19, 2025 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants