Description:

We successfully upgraded to 3.5.1 but when we later tried to restart Rocketchat it hung at SAUMonitor ➔ info [aggregate] - Start Cron. and progressed no further.

Since this was in Kubernetes the liveness probe eventually killed it, each time getting stuck at the same point until eventually after 7 times it managed to progress and start correctly. It's possible that it would've succeeded with a longer wait for liveness check but there's no way to verify now as it seems to be fine now.

See additional context for some background to why we restarted.

Steps to reproduce:

Not able to reproduce now, but it was upon start of a new kubernetes pod.

Expected behavior:

Rocketchat starts up in a timely fashion.

Actual behavior:

Rocketchat container starts and gets to a certain point in the logs (see logs) and hangs before eventually being killed by kubernetes due to liveness/readiness probes.

Server Setup Information:

• Version of Rocket.Chat Server: 3.5.1
• Operating System: Official docker image 3.5.1
• Deployment Method: docker in kubernetes
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version:
• MongoDB Version:

Client Setup Information

• Desktop App or Browser Version:
• Operating System:

Additional context

One point to note is the reason we restarted was to apply the enabling of push notifications, I'm not sure if there is any possible interaction here. We received the notice to enable them and actioned accordingly and we then encountered this.

These are the liveness/readiness checks in kubernetes which have served us well for a few years:

        livenessProbe:
          httpGet:
            path: /home
            port: 3000
          initialDelaySeconds: 90
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /home
            port: 3000
          initialDelaySeconds: 30
          timeoutSeconds: 10

Relevant logs:

kubectl -n xxxx logs rocketchat-7b4dc69ddd-4k77b
rocketchat:lib ➔ oauth_updated Accounts_OAuth_Facebook
...
...
rocketchat:lib ➔ oauth_updated Accounts_OAuth_Apple
Federation ➔ Setup.info Federation is disabled
Setting default file store to FileSystem
CAS ➔ info Disabling CAS login service
Search Logger ➔ info create search provider defaultProvider
Search Logger ➔ info create search provider chatpalProvider
{"line":"120","file":"migrations.js","message":"Migrations: Not migrating, already at version 202","time":{"$date":1596709208408},"level":"info"}
ufs: temp directory created at "/tmp/ufs"
SAUMonitor ➔ info [aggregate] - Start Cron.