What Is NDJSON? Streaming Records, Syntax, and Use Cases

Scrapeless Scraping API returns structured JSON that a downstream pipeline can frame as NDJSON when independent records need line-oriented streaming or storage.

What Is NDJSON?

NDJSON is a text format for a sequence of JSON values separated by newline characters. Each physical line contains one complete JSON text. A consumer reads a line, parses that line as JSON, processes the resulting value, and moves to the next record.

The NDJSON specification requires each JSON text to conform to JSON rules and to be followed by a newline. It specifies UTF-8, accepts LF and CRLF as line delimiters for parsing, and recommends application/x-ndjson with the .ndjson extension.

NDJSON is a framing convention around JSON, not a new object model. A line can technically contain any JSON value, although object-per-line is the dominant pattern for logs, bulk ingestion, exports, and data pipelines. The JSON grammar remains the one defined by RFC 8259.

What Does an NDJSON File Look Like?

A product observation stream might contain three independent records:

{"sku":"A-17","price":34.5,"currency":"USD"}
{"sku":"B-08","price":28,"currency":"USD"}
{"sku":"C-31","price":null,"currency":"EUR"}

There is no opening array bracket, comma between records, or closing bracket. Each line can be parsed on its own. The final line should end with a newline under the NDJSON serialization rule, although text viewers do not always make the final delimiter visible.

Pretty-printed JSON does not work as one NDJSON record because indentation writes a value across several physical lines. Producers should serialize each value in compact form. A string that contains a logical line break remains valid because JSON escapes it:

{"id":41,"message":"first line\nsecond line"}

The two characters backslash and n stay inside the JSON string on one physical line. A JSON parser reconstructs the line-break character after record framing has already succeeded.

NDJSON vs a JSON Array

Why NDJSON Works for Streaming

Standard JSON does not define a boundary between two adjacent top-level values. Writing {}{} leaves a parser without a standard separator. NDJSON assigns that role to the newline. The reader does not need to scan for balanced braces because braces inside JSON strings are ordinary string characters and the physical line boundary ends the record.

A producer can flush each line when the record is ready. A consumer can apply backpressure through its stream interface, parse one line, validate the value, and release memory after processing. This keeps memory use tied to the largest record and pipeline buffers rather than the entire dataset.

NDJSON is not the only JSON sequence format. RFC 7464 defines JSON text sequences using an ASCII record-separator character before each JSON text. That framing can tolerate pretty-printed values because record boundaries do not depend solely on line endings. Producers and consumers must agree on which sequence format they use.

NDJSON Record Design

A strong NDJSON stream gives each line enough context to be processed independently. Include a stable record type or schema version when several event shapes share one stream. Include an identifier that supports deduplication when the transport may deliver the same logical record more than once. Add event and observation times only with documented formats and time-zone semantics.

Keep large binary content out of line-oriented JSON unless the contract explicitly requires encoded bytes. Base64 increases size and creates very long records. A better event may carry a controlled object reference plus integrity metadata, subject to authorization at retrieval time.

Ordering must be explicit. NDJSON preserves physical line order, but distributed producers, partitions, and parallel consumers can change observed processing order. If order matters within an entity, include a sequence or version and define how gaps and out-of-order records are handled.

Schema Validation

Valid JSON is not necessarily a valid business record. A line may parse successfully while missing a required identifier or storing a number where the contract expects a string. Validate each parsed value against a record schema before using it.

Streams with multiple record types can choose a schema based on a stable discriminator. The dispatcher should reject unknown types or route them to a controlled quarantine path. Schema versions should define compatibility so consumers can continue when optional fields are added.

Record-level validation lets a batch report specific failures without losing the location of acceptable records. Store the physical line number, byte offset when available, schema error, and a safely redacted record identifier. Do not copy secrets or sensitive payloads into error logs.

Common NDJSON Use Cases

NDJSON, CSV, and Parquet

NDJSON preserves nested JSON structures and allows records with optional fields. CSV is more compact and approachable when every record is one flat table row. Parquet adds typed columnar storage for repeated analytics across many records.

A common pipeline collects or receives JSON, writes raw NDJSON for append-friendly traceability, validates and normalizes records, then publishes Parquet for analytical queries. CSV remains useful for selected flat exports to spreadsheet users. Each stage has a different consumer and therefore a different best format.

Compression and Splitting

Text records often compress well because keys and value patterns repeat. Whole-file compression reduces storage and transfer size, but some codecs make it difficult to begin reading from the middle of a compressed stream. Splittable compression or independently compressed chunks may be better for parallel processing.

Split only on complete record boundaries. A byte-range cut through the middle of a JSON string creates invalid fragments. Systems that need parallel access can maintain block indexes, chunk the stream into multiple objects, or use storage formats built for selective reads.

Concatenating valid NDJSON files usually preserves valid line framing when every input ends with a newline. If one file lacks the final delimiter, its last record can run into the first record of the next file. Writers should always terminate serialized records, including the final one.

Security and Operational Limits

Apply limits to total bytes, line length, nesting depth, string length, numeric magnitude, and allowed property count. A single NDJSON line can be arbitrarily large unless the application enforces a boundary. Read with a bounded buffer or streaming strategy that reports an oversized record without exhausting memory.

Do not execute fields as commands or templates. Escape values when they enter HTML, SQL, shell, or log contexts. Protect against log forging when NDJSON records are later converted into plain text. Keep authorization at the stream and record level when one file may contain data for several tenants.

How to Process NDJSON Reliably

1. Open the stream as UTF-8. Define how invalid byte sequences are reported; silent replacement can change identifiers.
2. Read one bounded physical line. Accept the agreed line endings and enforce a maximum record size.
3. Handle empty lines by contract. Decide whether they are ignored or rejected, and apply the rule consistently.
4. Parse one JSON value. Reject trailing non-whitespace content on that line and define duplicate-member behavior.
5. Validate the record schema. Check type, required properties, value limits, and supported versions.
6. Process idempotently when possible. Stable record identifiers help prevent duplicate side effects when a record appears more than once.
7. Record progress safely. Checkpoints should identify a durable record or byte boundary without claiming that an incomplete line was processed.

When Not to Use NDJSON

Use a normal JSON document when the payload is small, must carry top-level metadata, or benefits from pretty printing. Use CSV when the data is a flat table for spreadsheet consumers. Use Parquet when analytical engines need column pruning, typed storage, and compression across large datasets.

NDJSON is also a poor fit when individual values must contain unescaped physical line formatting for human editing. A record-separator-based JSON sequence or a framed binary protocol may match that requirement better.

Conclusion

NDJSON adds one practical rule to JSON exchange: each line is one complete JSON value. That rule supports append-friendly files, streaming parsers, record-level validation, and bounded memory. It does not define the business schema, ordering guarantees, security policy, or delivery semantics. A dependable NDJSON workflow uses compact UTF-8 records, explicit schemas, size limits, stable identifiers, clear empty-line behavior, and line-aware checkpoints.