Streaming JSON parser

So I found something I don’t like about nodejs.  It has a low memory limit.  The limit is around 2 GB of memory ; even on a 64 bit system.  It is a V8 thing.  If you don’t believe me try parsing a json file > 1GB – not going to work.  You end up with a nice out of memory exception.

So how do you parse a very large json file?  The same way you do everything else in NodeJS – async.  You stream the file and wait for events.  The jsonparse (https://github.com/creationix/jsonparse/) module is a huge help!

var Parser = require('jsonparse');

var fs = require('fs');

var p = new Parser();

p.onValue = function (value) {
 if(value.StatementNumber && value.StatementNumber == "12345"){
 console.log("Found me");
 }
};

var stream = fs.createReadStream('data.json', {});
stream.on('data',function(buffer){
 p.write(buffer);
});