JSON vs NDJSON: what happens when the file gets big

Both of these files hold three subscriber records. They contain almost the same bytes. Which one you pick makes no difference at all until the file gets big, or the job dies halfway through, or one record turns out to be malformed — and then it makes all the difference there is.

[
  { "msisdn": "447700900142", "iccid": "8944500102198765432", "status": "active" },
  { "msisdn": "447700900198", "iccid": "8944500102198765433", "status": "active" },
  { "msisdn": "447700900211", "iccid": "8944500102198765434", "status": "suspended" }
]

{"msisdn":"447700900142","iccid":"8944500102198765432","status":"active"}
{"msisdn":"447700900198","iccid":"8944500102198765433","status":"active"}
{"msisdn":"447700900211","iccid":"8944500102198765434","status":"suspended"}

NDJSON — newline-delimited JSON, also called JSON Lines and usually saved as .jsonl — has no brackets and no commas between records. That absence is the entire design. Every line is a complete, independent JSON document, and nothing outside a line has any bearing on it.

The wall you hit first: one array must be parsed whole

A JSON array is a single value. JSON.parse cannot hand you the first element until it has read the closing bracket, because until then it does not know the document is well formed. That is not an implementation shortcoming, it is what the grammar in RFC 8259 requires.

So the whole file has to fit in memory, twice over — once as the raw string and again as the parsed objects. On Node that ends at a specific, findable number:

$ node -e 'const fs=require("fs");JSON.parse(fs.readFileSync("subscribers.json","utf8"))'

buffer.js:  Cannot create a string longer than 0x1fffffe8 characters
            ^
RangeError: Invalid string length

0x1fffffe8 is 536,870,888 — about 512 MB. That is V8's maximum string length, exposed by Node as buffer.constants.MAX_STRING_LENGTH, and you meet it before you meet the heap limit because the file becomes a string before it becomes anything else. No flag raises it; --max-old-space-size does not help, since the ceiling is on the string, not the heap.

Now the same data as NDJSON. Each line is parsed on its own, so memory holds one record at a time regardless of whether the file is 600 MB or 600 GB:

$ node -e '
const rl = require("readline").createInterface({
  input: require("fs").createReadStream("subscribers.ndjson")
});
let active = 0;
rl.on("line", l => { if (JSON.parse(l).status === "active") active++; });
rl.on("close", () => console.log(active, "active"));
'
2841193 active

One bad record: does it cost you the file?

This is the difference that costs the most time, and the one people discover at the worst moment. Here is a file of 40,000 records where line 12,853 was truncated by a disk filling up mid-write.

$ node -e 'JSON.parse(require("fs").readFileSync("export.json","utf8"))'

SyntaxError: Unexpected token } in JSON at position 4718209

You get nothing. Not 39,999 records and a warning — nothing. The array is one value and one value is either valid or it is not. And the error names a byte offset, so before you can even see which record broke you have to work out what lives at position 4,718,209.

NDJSON has no such coupling. A broken line is a broken line:

$ node -e '
const lines = require("fs").readFileSync("export.ndjson","utf8").split("\n");
let ok = 0; const bad = [];
lines.forEach((l, i) => {
  if (!l.trim()) return;
  try { JSON.parse(l); ok++; } catch { bad.push(i + 1); }
});
console.log(ok, "parsed,", bad.length, "failed at line", bad.join(", "));
'
39999 parsed, 1 failed at line 12853

39,999 usable records and a line number you can go and look at. This is why our own NDJSON to Table parses each line independently and reports failures with their original line numbers rather than stopping at the first one — anything else would throw away the property that makes the format worth using.

Appending, and why it is the quiet decider

Adding one record to an NDJSON file is opening it in append mode and writing a line. Adding one record to a JSON array means either loading and rewriting the whole document, or seeking backwards past the closing bracket and splicing — which works until two processes do it at once.

# NDJSON: atomic for lines under the pipe buffer, safe from several writers
echo '{"msisdn":"447700900244","status":"active"}' >> subscribers.ndjson

# JSON array: read 600 MB, parse it, push, serialise, write 600 MB back
node -e '...'    # and hope nothing else is writing

This is why every append-only system that has to write JSON ends up at NDJSON regardless of what it started with. Log shippers, event streams, and the export endpoints of most APIs all emit it. It is also why docker logs, BigQuery's JSON loader and Elasticsearch's bulk API all take newline-delimited input rather than arrays.

What NDJSON gives up

It costs you real things, and pretending otherwise is how you end up using it for a 40-line config file.

• You cannot pretty-print it. Indentation puts a record across several lines, and in NDJSON a newline ends the record. A minified line is the format. That makes a raw NDJSON file genuinely unpleasant to read by eye, which is the main reason people convert it to something else to look at it.
• There is no place for top-level metadata. A JSON response can wrap its records in { "total": 4000, "page": 2, "results": [...] }. NDJSON has no envelope — every line is a record, so pagination and counts have to travel somewhere else, usually in HTTP headers.
• It is not a single JSON document. Anything expecting one — a schema validator pointed at the file, a config loader, JSON.parse — will reject it outright. This surprises people constantly.
• The name is not settled. NDJSON, JSON Lines, JSONL, "line-delimited JSON" and application/x-ndjson all mean the same thing, which makes searching for tooling more annoying than it should be.

Something both formats get wrong together

Neither format protects you from the precision problem, and it is worth saying plainly because it is easy to assume the newer one fixed it. Both of these lose the same digits:

> JSON.parse('{"iccid": 8944500102198765432}').iccid
8944500102198766000

> JSON.parse('{"iccid":8944500102198765432}').iccid      // one NDJSON line
8944500102198766000

The parser is the same in both cases, so the rounding is the same. An ICCID is 19 or 20 digits and a JavaScript number holds about 15 reliably — the fix is to keep the value as a string in the source, or to parse in a way that preserves the digits. The long version of this covers which identifiers it destroys and why the correct fix differs depending on whether you are displaying the value or re-emitting it.

Picking one

Three questions, in the order that actually decides it.

1. Will this file ever be appended to? If yes, NDJSON. Nothing else in this comparison matters as much, because the alternative is rewriting the file on every write.
2. Could it outgrow memory? An export, a log, an event stream, anything whose size is set by how long it runs rather than by what it describes — NDJSON. A configuration file, an API response, a document describing one thing — an array is fine and reads better.
3. Does a consumer need it whole? A schema validator, a config loader, a browser fetch().json() — those want one document. Give them an array.

The most common right answer in practice is both, at different points: NDJSON on disk and over the wire where size and appending matter, converted to an array at the edge where something needs the whole thing. Converting is cheap and lossless in both directions — NDJSON to JSON and JSON to NDJSON do it in the browser, and NDJSON to Table is usually what you want when the real question is just "what is in this file?".

If you remember three things

• A JSON array is one value. It has to be parsed whole, so one bad record costs you all of them and a big file costs you a RangeError at about 512 MB in Node.
• NDJSON trades readability and the envelope for independence: stream it, append to it, split it across workers, lose only the line that is broken.
• Neither of them saves your 19-digit identifiers. That is a JavaScript number problem, not a format problem, and it needs a deliberate fix in both.