Streaming JSON: How to Handle Large Datasets Without Running Out of Memory

What happens when you need to process a JSON file that's 500MB or larger? Or a continuous stream of JSON events from a WebSocket or event bus? Loading it all into memory with JSON.parse() will crash your process. Streaming JSON is the answer.

Why Standard JSON.parse() Fails for Large Data

JSON.parse() loads the entire JSON string into memory at once. For a 100MB file, Node.js may need 3–5x that in heap memory during parsing. A 500MB file can easily exhaust available RAM and trigger an out-of-memory crash.

Streaming parsers read JSON token by token, emitting events as they encounter keys, values, array items, and objects — without ever holding the full document in memory.

Node.js: JSONStream

JSONStream is the classic Node.js streaming JSON parser. It integrates with Node.js readable/writable streams.

npm install JSONStream

const fs = require('fs');
const JSONStream = require('JSONStream');

const fileStream = fs.createReadStream('./large-data.json');
const jsonStream = JSONStream.parse('items.*'); // stream each item in the "items" array

jsonStream.on('data', (item) => {
  console.log('Processing:', item.id); // handle one item at a time
});

jsonStream.on('end', () => {
  console.log('Done processing all items.');
});

fileStream.pipe(jsonStream);

This approach uses near-constant memory regardless of file size, because each item is processed and garbage-collected before the next one is read.

Node.js: stream-json

For more advanced use-cases (nested streaming, object assembly, filtering), stream-json is a more modern and powerful option.

npm install stream-json

const { chain } = require('stream-chain');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');
const fs = require('fs');

const pipeline = chain([
  fs.createReadStream('./large-data.json'),
  parser(),
  streamArray(),
]);

pipeline.on('data', ({ key, value }) => {
  // key = array index, value = one object from the array
  processRecord(value);
});

Python: ijson

In Python, the ijson library provides iterative JSON parsing:

pip install ijson

import ijson

with open('large-data.json', 'rb') as f:
    for item in ijson.items(f, 'items.item'):
        process(item)  # one item at a time, minimal memory

NDJSON — Newline-Delimited JSON

For the highest streaming performance, consider NDJSON (Newline-Delimited JSON). Instead of one giant array, each line is a separate JSON object:

{"id": 1, "name": "Alice"}
{"id": 2, "name": "Bob"}
{"id": 3, "name": "Carol"}

This format is trivially streamable — you can read line by line with a simple readline interface and parse each line independently:

const readline = require('readline');
const fs = require('fs');

const rl = readline.createInterface({ input: fs.createReadStream('./data.ndjson') });

rl.on('line', (line) => {
  const record = JSON.parse(line);
  processRecord(record);
});

NDJSON is widely used in logging, big data pipelines (used by Elasticsearch, AWS CloudTrail), and any system that emits continuous JSON events.

WebSocket: Real-Time JSON Streams

For real-time applications, WebSockets deliver continuous JSON messages:

const ws = new WebSocket('wss://api.example.com/stream');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  updateUI(data); // handle each message as it arrives
};

Each message is a small, self-contained JSON payload — no streaming parser needed since each message is already a complete, bounded JSON string.

When to Use Each Approach

Streaming JSON is one of those techniques that you rarely need — until you do, at which point it becomes absolutely critical. Adding it to your toolkit means you're ready to handle datasets of any size without running into memory walls.