[1.6.1] 100% CPU on Ubuntu server, caused by client? #9796
Description
We are experiencing consistent 100% CPU usage on one of our Meteor deployments. This issue seems to have appeared out of nowhere (not after a of a new version).
What we already tried: create a new server with the same specs and deploy to that server. When we switch over our DNS to the new server at first all is well; but (we think) when a particular client connects the Node process starts using 100% CPU.
We're deploying Meteor to a Ubuntu host using MUP (which instantiates a Docker container consisting of the meteord base image and our bundle). The image has NodeJS 8.9.4 and NPM 5.6.0.
Any pointers as to how to pinpoint this issue would be greatly appreciated!
We believe this is the most interesting portion of the V8 profiler we ran on the logs (https://paste2.org/zPsHbDya):
ticks parent name
2420054 92.5% /opt/nodejs/bin/node
219101 9.1% /opt/nodejs/bin/node
165262 75.4% Builtin: ArrayMap
57108 34.6% LazyCompile: *changeField /bundle/bundle/programs/server/packages/ddp-server.js:287:25
57067 99.9% LazyCompile: *added /bundle/bundle/programs/server/packages/mongo.js:3650:23
57042 100.0% LazyCompile: *<anonymous> packages/meteor.js:1231:19
48280 29.2% LazyCompile: *EJSON.clone.v /bundle/bundle/programs/server/packages/ejson.js:646:15
47783 99.0% Function: ~Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:697:26
47783 100.0% Builtin: ArrayForEach
23827 14.4% LazyCompile: *Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:320:28
23616 99.1% LazyCompile: *added /bundle/bundle/programs/server/packages/mongo.js:3650:23
23616 100.0% LazyCompile: *<anonymous> packages/meteor.js:1231:19
22206 13.4% LazyCompile: *Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:697:26
22205 100.0% Builtin: ArrayForEach
15641 70.4% LazyCompile: *_sendAdds /bundle/bundle/programs/server/packages/mongo.js:1913:23
3405 15.3% LazyCompile: *EJSON.clone.v /bundle/bundle/programs/server/packages/ejson.js:646:15
2442 11.0% LazyCompile: *<anonymous> /bundle/bundle/programs/server/packages/mongo.js:1782:34
6566 4.0% Builtin: ArrayForEach
4420 67.3% LazyCompile: *_sendAdds /bundle/bundle/programs/server/packages/mongo.js:1913:23
4420 100.0% Function: ~<anonymous> /bundle/bundle/programs/server/packages/mongo.js:1782:34
2082 31.7% LazyCompile: *<anonymous> /bundle/bundle/programs/server/packages/mongo.js:1782:34
2082 100.0% LazyCompile: *<anonymous> packages/meteor.js:1231:19
3827 2.3% RegExp: ([!#\\$%&'\\*\\+\\-\\.\\^_`\\|~0-9a-z]+)(?:=(?:([!#\\$%&'\\*\\+\\-\\.\\^_`\\|~0-9a-z]+)|"((?:\\\\[\\x00-\\x7f]|[^\\x00-\\x08\\x0a-\\x1f\\x7f"])*)"))? {9}
3812 99.6% Function: ~Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:697:26
3812 100.0% Builtin: ArrayForEach
25601 11.7% LazyCompile: *ObserveHandle.stop /bundle/bundle/programs/server/packages/mongo.js:1955:41
21756 85.0% Function: ~<anonymous> /bundle/bundle/programs/server/packages/mongo.js:3663:25
21756 100.0% LazyCompile: *<anonymous> packages/meteor.js:1231:19
19776 90.9% Function: ~<anonymous> /bundle/bundle/programs/server/packages/ddp-server.js:1298:32
1978 9.1% LazyCompile: *<anonymous> /bundle/bundle/programs/server/packages/ddp-server.js:1298:32
2805 11.0% LazyCompile: *baseUniq /bundle/bundle/programs/server/npm/node_modules/lodash.union/index.js:742:18
2805 100.0% LazyCompile: *<anonymous> packages/meteor.js:1231:19
2805 100.0% Function: ~<anonymous> /bundle/bundle/programs/server/packages/ddp-server.js:1298:32
1040 4.1% LazyCompile: *<anonymous> /bundle/bundle/programs/server/packages/mongo.js:3663:25
1040 100.0% LazyCompile: *<anonymous> packages/meteor.js:1231:19
535 51.4% Function: ~<anonymous> /bundle/bundle/programs/server/packages/ddp-server.js:1298:32
505 48.6% LazyCompile: *<anonymous> /bundle/bundle/programs/server/packages/ddp-server.js:1298:32
2944 1.3% LazyCompile: *v.map.value /bundle/bundle/programs/server/packages/ejson.js:678:18
2607 88.6% Function: ~Socket._writeGeneric net.js:708:42
2607 100.0% LazyCompile: *Socket._write net.js:785:35
1686 64.7% LazyCompile: *ondata internal/streams/legacy.js:14:18
495 19.0% LazyCompile: *Connection.write /bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:527:38
419 16.1% LazyCompile: *clearBuffer _stream_writable.js:469:21
179 6.1% Builtin: ArrayMap
57 31.8% LazyCompile: *EJSON.clone.v /bundle/bundle/programs/server/packages/ejson.js:646:15
52 91.2% Function: ~Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:697:26
4 7.0% Function: ~<anonymous> /bundle/bundle/programs/server/packages/mongo.js:1879:36
1 1.8% Function: ~changeField /bundle/bundle/programs/server/packages/ddp-server.js:287:25
55 30.7% LazyCompile: *changeField /bundle/bundle/programs/server/packages/ddp-server.js:287:25
55 100.0% LazyCompile: *added /bundle/bundle/programs/server/packages/mongo.js:3650:23
31 17.3% LazyCompile: *Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:697:26
31 100.0% Builtin: ArrayForEach
20 11.2% LazyCompile: *Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:320:28
20 100.0% LazyCompile: *added /bundle/bundle/programs/server/packages/mongo.js:3650:23
5 2.8% Builtin: ArrayForEach
5 100.0% LazyCompile: *_sendAdds /bundle/bundle/programs/server/packages/mongo.js:1913:23
4 2.2% RegExp: ([!#\\$%&'\\*\\+\\-\\.\\^_`\\|~0-9a-z]+)(?:=(?:([!#\\$%&'\\*\\+\\-\\.\\^_`\\|~0-9a-z]+)|"((?:\\\\[\\x00-\\x7f]|[^\\x00-\\x08\\x0a-\\x1f\\x7f"])*)"))? {9}
4 100.0% Function: ~Object.keys.forEach.key /bundle/bundle/programs/server/packages/ejson.js:697:26
4 2.2% LazyCompile: *<anonymous> /bundle/bundle/programs/server/packages/mongo.js:1879:36
4 100.0% Function: ~SQp._run packages/meteor.js:851:21
3 1.7% LazyCompile: *callback zlib.js:447:20
3 100.0% Builtin: ArrayForEach
60 2.0% Function: ~<anonymous> /bundle/bundle/programs/server/packages/ddp-server.js:740:22