{"id":19356,"date":"2013-12-03T13:00:31","date_gmt":"2013-12-03T11:00:31","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/?p=19356"},"modified":"2013-12-02T22:22:34","modified_gmt":"2013-12-02T20:22:34","slug":"mongodb-facts-80000-insertssecond-on-commodity-hardware","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2013\/12\/mongodb-facts-80000-insertssecond-on-commodity-hardware.html","title":{"rendered":"MongoDB Facts: 80000+ inserts\/second on commodity hardware"},"content":{"rendered":"

While experimenting with some time series collections I needed a large data set to check that our aggregation queries don\u2019t become a bottleneck in case of increasing data loads. We settled for 50 million documents, since beyond this number we would consider sharding anyway.<\/p>\n

Each time event looks like this:
\n 
\n 
\n 
\n 
\n <\/p>\n

{\r\n        \"_id\" : ObjectId(\"5298a5a03b3f4220588fe57c\"),\r\n        \"created_on\" : ISODate(\"2012-04-22T01:09:53Z\"),\r\n        \"value\" : 0.1647851116706831\r\n}<\/pre>\n
As we wanted to get random values, we thought of generating them using JavaScript or Python (we could have tried in in Java, but we wanted to write it as fast as possible). We didn\u2019t know which one will be faster so we decided to test them.<\/p>\n
Our first try was with a JavaScript file run through the MongoDB shell.<\/p>\n
Here is how it looks like:<\/p>\n
var minDate = new Date(2012, 0, 1, 0, 0, 0, 0);\r\nvar maxDate = new Date(2013, 0, 1, 0, 0, 0, 0);\r\nvar delta = maxDate.getTime() - minDate.getTime();\r\n\r\nvar job_id = arg2;\r\n\r\nvar documentNumber = arg1;\r\nvar batchNumber = 5 * 1000;\r\n\r\nvar job_name = 'Job#' + job_id\r\nvar start = new Date();\r\n\r\nvar batchDocuments = new Array();\r\nvar index = 0;\r\n\r\nwhile(index < documentNumber) {\r\n\tvar date = new Date(minDate.getTime() + Math.random() * delta);\r\n\tvar value = Math.random();\r\n\tvar document = {\t\t\r\n\t\tcreated_on : date,\r\n\t\tvalue : value\r\n\t};\r\n\tbatchDocuments[index % batchNumber] = document;\r\n\tif((index + 1) % batchNumber == 0) {\r\n\t\tdb.randomData.insert(batchDocuments);\r\n\t}\r\n\tindex++;\r\n\tif(index % 100000 == 0) {\t\r\n\t\tprint(job_name + ' inserted ' + index + ' documents.');\r\n\t}\r\n}\r\nprint(job_name + ' inserted ' + documentNumber + ' in ' + (new Date() - start)\/1000.0 + 's');<\/pre>\n
This is how we run it and what we got:<\/p>\n
mongo random --eval \"var arg1=50000000;arg2=1\" create_random.js\r\nJob#1 inserted 100000 documents.\r\nJob#1 inserted 200000 documents.\r\nJob#1 inserted 300000 documents.\r\n...\r\nJob#1 inserted 49900000 documents.\r\nJob#1 inserted 50000000 in 566.294s<\/pre>\n
Well, this is beyond my wild expectations already (88293 inserts\/second).<\/p>\n
Now it\u2019s Python\u2019s turn. You will need to install pymongo to properly run it.<\/p>\n
import sys\r\nimport os\r\nimport pymongo\r\nimport time\r\nimport random\r\n\r\nfrom datetime import datetime\r\n\r\nmin_date = datetime(2012, 1, 1)\r\nmax_date = datetime(2013, 1, 1)\r\ndelta = (max_date - min_date).total_seconds()\r\n\r\njob_id = '1'\r\n\r\nif len(sys.argv) < 2:\r\n\tsys.exit(\"You must supply the item_number argument\")\r\nelif len(sys.argv) > 2:\r\n\tjob_id = sys.argv[2]\t\r\n\r\ndocuments_number = int(sys.argv[1])\r\nbatch_number = 5 * 1000;\r\n\r\njob_name = 'Job#' + job_id\r\nstart = datetime.now();\r\n\r\n# obtain a mongo connection\r\nconnection = pymongo.Connection(\"mongodb:\/\/localhost\", safe=True)\r\n\r\n# obtain a handle to the random database\r\ndb = connection.random\r\ncollection = db.randomData\r\n\r\nbatch_documents = [i for i in range(batch_number)];\r\n\r\nfor index in range(documents_number):\r\n\ttry:\t\t\t\r\n\t\tdate = datetime.fromtimestamp(time.mktime(min_date.timetuple()) + int(round(random.random() * delta)))\r\n\t\tvalue = random.random()\r\n\t\tdocument = {\r\n\t\t\t'created_on' : date,\t\r\n\t\t\t'value' : value,\t\r\n\t\t}\r\n\t\tbatch_documents[index % batch_number] = document\r\n\t\tif (index + 1) % batch_number == 0:\r\n\t\t\tcollection.insert(batch_documents)\t\t\r\n\t\tindex += 1;\r\n\t\tif index % 100000 == 0:\t\r\n\t\t\tprint job_name, ' inserted ', index, ' documents.'\t\t\r\n\texcept:\r\n\t\tprint 'Unexpected error:', sys.exc_info()[0], ', for index ', index\r\n\t\traise\r\nprint job_name, ' inserted ', documents_number, ' in ', (datetime.now() - start).total_seconds(), 's'<\/pre>\n
We run it and this is what we got this time:
 
<\/div> <\/div><\/p>\n
python create_random.py 50000000\r\nJob#1  inserted  100000  documents.\r\nJob#1  inserted  200000  documents.\r\nJob#1  inserted  300000  documents.\r\n...\r\nJob#1  inserted  49900000  documents.\r\nJob#1  inserted  50000000  in  1713.501 s<\/pre>\n
This is slower compared to the JavaScript version (29180 inserts\/second), but lets not get discouraged. Python is a full-featured programming language, so how about taking advantage of all our CPU cores (e.g. 4 cores) and start one script per core, each one inserting a fraction of the total documents number (e.g. 12500000).<\/p>\n
import sys\r\nimport pymongo\r\nimport time\r\nimport subprocess\r\nimport multiprocessing\r\n\r\nfrom datetime import datetime\r\n\r\ncpu_count = multiprocessing.cpu_count()\r\n\r\n# obtain a mongo connection\r\nconnection = pymongo.Connection('mongodb:\/\/localhost', safe=True)\r\n\r\n# obtain a handle to the random database\r\ndb = connection.random\r\ncollection = db.randomData\r\n\r\ntotal_documents_count = 50 * 1000 * 1000;\r\ninserted_documents_count = 0\r\nsleep_seconds = 1\r\nsleep_count = 0\r\n\r\nfor i in range(cpu_count):\r\n\tdocuments_number = str(total_documents_count\/cpu_count)\r\n\tprint documents_number\r\n\tsubprocess.Popen(['python', '..\/create_random.py', documents_number, str(i)])\r\n\r\nstart = datetime.now();\r\n\r\nwhile (inserted_documents_count < total_documents_count) is True:\r\n\tinserted_documents_count = collection.count()\r\n\tif (sleep_count > 0 and sleep_count % 60 == 0):\t\r\n\t\tprint 'Inserted ', inserted_documents_count, ' documents.'\t\t\r\n\tif (inserted_documents_count < total_documents_count):\r\n\t\tsleep_count += 1\r\n\t\ttime.sleep(sleep_seconds)\t\r\n\r\nprint 'Inserting ', total_documents_count, ' took ', (datetime.now() - start).total_seconds(), 's'<\/pre>\n
Running the parallel execution Python script goes like this:<\/p>\n
python create_random_parallel.py\r\nJob#3  inserted  100000  documents.\r\nJob#2  inserted  100000  documents.\r\nJob#0  inserted  100000  documents.\r\nJob#1  inserted  100000  documents.\r\nJob#3  inserted  200000  documents.\r\n...\r\nJob#2  inserted  12500000  in  571.819 s\r\nJob#0  inserted  12400000  documents.\r\nJob#3  inserted  10800000  documents.\r\nJob#1  inserted  12400000  documents.\r\nJob#0  inserted  12500000  documents.\r\nJob#0  inserted  12500000  in  577.061 s\r\nJob#3  inserted  10900000  documents.\r\nJob#1  inserted  12500000  documents.\r\nJob#1  inserted  12500000  in  578.427 s\r\nJob#3  inserted  11000000  documents.\r\n...\r\nJob#3  inserted  12500000  in  623.999 s\r\nInserting  50000000  took  624.655 s<\/pre>\n
This is very good indeed (80044 inserts\/seconds) even if still slower than the first JavaScript import. So lets adapt this last Python script to run the JavaScript through multiple MongoDB shells.<\/p>\n
Since I couldn\u2019t supply the required arguments to the mongo command, to the sub-process started by the main python script, I came up with the following alternative:<\/p>\n
for i in range(cpu_count):\r\n\tdocuments_number = str(total_documents_count\/cpu_count)\r\n\tscript_name = 'create_random_' + str(i + 1) + '.bat'\r\n\tscript_file = open(script_name, 'w')\r\n\tscript_file.write('mongo random --eval \"var arg1=' + documents_number +';arg2=' + str(i + 1) +'\" ..\/create_random.js');\r\n\tscript_file.close()\r\n\tsubprocess.Popen(script_name)<\/pre>\n
We generate shell scripts dynamically and let python run them for us.<\/p>\n
Job#1 inserted 100000 documents.\r\nJob#4 inserted 100000 documents.\r\nJob#3 inserted 100000 documents.\r\nJob#2 inserted 100000 documents.\r\nJob#1 inserted 200000 documents.\r\n...\r\nJob#4 inserted 12500000 in 566.438s\r\nJob#3 inserted 12300000 documents.\r\nJob#2 inserted 10800000 documents.\r\nJob#1 inserted 11600000 documents.\r\nJob#3 inserted 12400000 documents.\r\nJob#1 inserted 11700000 documents.\r\nJob#2 inserted 10900000 documents.\r\nJob#1 inserted 11800000 documents.\r\nJob#3 inserted 12500000 documents.\r\nJob#3 inserted 12500000 in 574.782s\r\nJob#2 inserted 11000000 documents.\r\nJob#1 inserted 11900000 documents.\r\nJob#2 inserted 11100000 documents.\r\nJob#1 inserted 12000000 documents.\r\nJob#2 inserted 11200000 documents.\r\nJob#1 inserted 12100000 documents.\r\nJob#2 inserted 11300000 documents.\r\nJob#1 inserted 12200000 documents.\r\nJob#2 inserted 11400000 documents.\r\nJob#1 inserted 12300000 documents.\r\nJob#2 inserted 11500000 documents.\r\nJob#1 inserted 12400000 documents.\r\nJob#2 inserted 11600000 documents.\r\nJob#1 inserted 12500000 documents.\r\nJob#1 inserted 12500000 in 591.073s\r\nJob#2 inserted 11700000 documents.\r\n...\r\nJob#2 inserted 12500000 in 599.005s\r\nInserting  50000000  took  599.253 s<\/pre>\n
This is fast too (83437 inserts\/second) but still can\u2019t beat our first attempt.<\/p>\n
Conclusion<\/h2>\n
My PC configuration is nothing out of the ordinary, and the only optimization is that I have a SSD drive on which MongoDB runs.<\/p>\n
\"vlad_pc\"<\/a><\/p>\n
The first attempt yielded the best results, and monitoring CPU resources I realized MongoDB leverages all of them even for a single shell console. The Python script running on all cores was also fast enough and it has the advantage of allowing us to turn this script into a fully operational application, if we want to.<\/p>\n