Skip to content

DB Cleanup causes New Device events #777

@nareddyt

Description

@nareddyt

Is there an existing issue for this?

Current Behavior

Whenever the DB Cleanup plugin runs, all devices get detected as "New Device". This causes the session graphs to look quite weird and generates a lot of extra events.

Screenshot 2024-08-29 at 3 09 55 PM
Screenshot 2024-08-29 at 3 20 13 PM

Expected Behavior

Pre-existing devices should not be marked as "New Device" during DB cleanup.

Steps To Reproduce

  1. Run NetAlertX 24.7.18 with default settings
  2. Notice every time the DB cleanup occurs, New Device events are generated for pre-existing devices.
  3. Other scheduled plugin scans that don't require a DB cleanup doesn't result in the new device list.
  4. Modify DB Cleanup cron schedule from default 30m to 15m and confirm New Device events occur every 15m now. This proves the "New Device" events are caused by DB Cleanup plugin.

Screenshot 2024-08-29 at 3 17 24 PM

app.conf

Pasted screenshot of settings page instead

docker-compose.yml

netalertx:
    image: "jokobsk/netalertx:24.7.18"
    container_name: netalertx
    network_mode: "host"
    restart: unless-stopped
    volumes:
      - ${HOME_DIR}/netalertx:/app/config
      - netalertx_data:/home/pi/netalertx/db
    environment:
      - TZ=${TZ}
      - PORT=20398

What branch are you running?

Production

app.log

15:31:28 [Scheduler] - Scheduler run for DBCLNP: YES
15:31:28 [Plugin utils] ---------------------------------------------
15:31:28 [Plugin utils] display_name: DB cleanup
15:31:28 [Plugins] CMD: python3 /app/front/plugins/db_cleanup/script.py pluginskeephistory={pluginskeephistory} hourstokeepnewdevice={hourstokeepnewdevice} daystokeepevents={daystokeepevents} pholuskeepdays={pholuskeepdays}
15:31:28 [Plugins] Resolving param: {'name': 'pluginskeephistory', 'type': 'setting', 'value': 'PLUGINS_KEEP_HIST'}
15:31:28 [Plugins] setTyp: {"dataType":"integer", "elements": [{"elementType" : "input", "elementOptions" : [{"type": "number"}] ,"transformers": []}]}
15:31:28 [Plugins] setTypJSN: {'dataType': 'integer', 'elements': [{'elementType': 'input', 'elementOptions': [{'type': 'number'}], 'transformers': []}]}
15:31:28 [Plugins] dType: integer
15:31:28 [Plugins] Resolved value: 250
15:31:28 [Plugins] Convert to Base64: False
15:31:28 [Plugins] Resolving param: {'name': 'daystokeepevents', 'type': 'setting', 'value': 'DAYS_TO_KEEP_EVENTS'}
15:31:28 [Plugins] setTyp: {"dataType":"integer", "elements": [{"elementType" : "input", "elementOptions" : [{"type": "number"}] ,"transformers": []}]}
15:31:28 [Plugins] setTypJSN: {'dataType': 'integer', 'elements': [{'elementType': 'input', 'elementOptions': [{'type': 'number'}], 'transformers': []}]}
15:31:28 [Plugins] dType: integer
15:31:28 [Plugins] Resolved value: 180
15:31:28 [Plugins] Convert to Base64: False
15:31:28 [Plugins] Resolving param: {'name': 'hourstokeepnewdevice', 'type': 'setting', 'value': 'HRS_TO_KEEP_NEWDEV'}
15:31:28 [Plugins] setTyp: {"dataType":"integer", "elements": [{"elementType" : "input", "elementOptions" : [{"type": "number"}] ,"transformers": []}]}
15:31:28 [Plugins] setTypJSN: {'dataType': 'integer', 'elements': [{'elementType': 'input', 'elementOptions': [{'type': 'number'}], 'transformers': []}]}
15:31:28 [Plugins] dType: integer
15:31:28 [Plugins] Resolved value: 168
15:31:28 [Plugins] Convert to Base64: False
15:31:28 [Plugins] Timeout: 30
15:31:28 [Plugin utils] Pre-Resolved CMD: python3/app/front/plugins/db_cleanup/script.pypluginskeephistory={pluginskeephistory}hourstokeepnewdevice={hourstokeepnewdevice}daystokeepevents={daystokeepevents}pholuskeepdays={pholuskeepdays}
15:31:28 [Plugins] Executing: python3 /app/front/plugins/db_cleanup/script.py pluginskeephistory={pluginskeephistory} hourstokeepnewdevice={hourstokeepnewdevice} daystokeepevents={daystokeepevents} pholuskeepdays={pholuskeepdays}
15:31:28 [Plugins] Resolved : ['python3', '/app/front/plugins/db_cleanup/script.py', 'pluginskeephistory=250', 'hourstokeepnewdevice=168', 'daystokeepevents=180', 'pholuskeepdays={pholuskeepdays}']
15:31:28 [DBCLNP] In script
15:31:28 [DBCLNP] Upkeep Database:
15:31:28 [DBCLNP] Online_History: Delete all but keep latest 150 entries
15:31:28 [DBCLNP] Events: Delete all older than 180 days (DAYS_TO_KEEP_EVENTS setting)
15:31:28 [DBCLNP] Plugins_History: Trim Plugins_History entries to less than 250 per Plugin (PLUGINS_KEEP_HIST setting)
15:31:28 [DBCLNP] Plugins_History: Trim Notifications entries to less than 100
15:31:28 [DBCLNP] Trim AppEvents to less than 5000
15:31:29 [DBCLNP] Devices: Delete all New Devices older than 168 hours (HRS_TO_KEEP_NEWDEV setting)
15:31:29 [DBCLNP] Pholus_Scan: Delete all older than 30 days (PHOLUS_DAYS_DATA setting)
15:31:29 [DBCLNP] Pholus_Scan: Delete all duplicates
15:31:29 [DBCLNP] Plugins_Objects: Delete all duplicates
15:31:29 [DBCLNP] Shrink Database
15:31:29 [DBCLNP] Cleanup complete
15:31:29 [Plugins] No output received from the plugin DBCLNP - enable LOG_LEVEL=debug and check logs
15:31:29 [Scheduler] - Scheduler run for MAINT: NO
15:31:29 [Scheduler] - Scheduler run for PHOLUS: NO
15:31:29 [Scheduler] - Scheduler run for VNDRPDT: NO
15:31:29 [Plugins] Check if any plugins need to be executed on run type: always_after_scan
15:31:29 [MAIN] processScan: True
15:31:29 [MAIN] start processig scan results
15:31:29 [Process Scan] Processing scan results
15:31:29 [Save Devices] Saving this IP into the CurrentScan table:192.168.0.101
15:31:29 [Process Scan] Print Stats
15:31:29 [Scan Stats] Devices Detected.......: 72
15:31:29 [Scan Stats] New Devices............: 71
15:31:29 [Scan Stats] Down Alerts............: 0
15:31:29 [Scan Stats] New Down Alerts........: 0
15:31:29 [Scan Stats] New Connections........: 0
15:31:29 [Scan Stats] Disconnections.........: 0
15:31:29 [Scan Stats] IP Changes.............: 0
15:31:29 ================ DEVICES table content ================

15:31:29 ================ Events table COUNT ================
15:31:29 {'count(*)': 3133}
15:31:29 [Scan Stats] Scan Method Statistics:
15:31:29 INTRNT: 1
15:31:29 UNFIMP: 41
15:31:29 arp-scan: 30
15:31:29 [Process Scan] Stats end
15:31:29 [Process Scan] Sessions Events (connect / discconnect)
15:31:29 [Events] - 1 - Devices down
15:31:29 [Events] - 2 - New Connections
15:31:29 [Events] - 3 - Disconnections
15:31:29 [Events] - 4 - IP Changes
15:31:29 [Events] - Events end
15:31:29 [Process Scan] Creating new devices
15:31:29 [New Devices] New devices - 1 Events

Debug enabled

  • I have read and followed the steps in the wiki link above and provided the required debug logs and the log section covers the time when the issue occurs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug 🐛Something isn't workingnext release/in dev image 🚀This is coming in the next release or was already released if the issue is Closed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions