Skip to content

Long MongoDB queries - performance #22193

@emikolajczak

Description

@emikolajczak

Description:

Hi, after upgrade to 3.14.0 I see in MongoDB logs a lots notifications about long last queries as below example

2021-05-28T10:46:50.657+0200 I COMMAND [conn322090] command rocketchat.rocketchat_message command: find { find: "rocketchat_message", filter: { _hidden: { $ne: true }, rid: "pwcbshayJiYEzQhJt", tcount: { $exists: true } }, sort: { tlm: -1 }, projection: { joinCode: 0, members: 0, importIds: 0, e2e: 0 }, limit: 25, returnKey: false, showRecordId: false, lsid: { id: UUID("13b58f4a-bb53-4300-8715-dc11ddc145dc") }, $clusterTime: { clusterTime: Timestamp(1622191610, 53), signature: { hash: BinData(0, B4E712374B9B6A2DF772871112E9BFFB109EB878), keyId: 6945063607709204546 } }, $db: "rocketchat" } planSummary: IXSCAN { tcount: 1, tlm: 1 } keysExamined:43586 docsExamined:43586 hasSortStage:1 cursorExhausted:1 numYields:340 nreturned:25 reslen:14672 locks:{ Global: { acquireCount: { r: 341 } }, Database: { acquireCount: { r: 341 } }, Collection: { acquireCount: { r: 341 } } } storage:{} protocol:op_msg 356ms

Before upgrade, on 3.9.7 version no entries about this query. I think it may be a cause heaviest load on DB.

Also graph for Meteor facts looks strange. In time this jumps of mongo-livedata time-spent-in-QUERYING-phase is see slow down of Rocket.Chat (greyed out messages i.e.)

image

EDIT:

From time to time you can observe spikes in Meteor facts

image

Let me know if you need more graphs or data for analysis.

Steps to reproduce:

  1. Check MongoDb logs for long processing queries
  2. Check Meteor facts graph in Metrics

Expected behavior:

No long processing operations in Db logs

Actual behavior:

A lot of long processing queries

Server Setup Information:

  • Version of Rocket.Chat Server: 3.14.0
  • Operating System: Centos7
  • Deployment Method: docker-compose
  • Number of Running Instances: 33
  • DB Replicaset Oplog: YES
  • NodeJS Version: v12.22.1
  • MongoDB Version: 4.0.14

Client Setup Information

  • Desktop App or Browser Version: all
  • Operating System: Windows 10

Additional context

Relevant logs:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions