JSON Lines and Newline-Delimited Imports

JSON Lines, often called NDJSON, stores one complete JSON value per line. It is common in exports, logs, queues, bulk imports, analytics events, data handoffs between systems, and command-line tools that stream records from one process to another.

The key advantage is streaming. PHP can read one line, decode one record, validate it, process it, and move on. It does not need to load a huge JSON array before work can begin. That makes JSON Lines useful when a file may contain many records, when you want progress reporting, or when you want to report line-level errors back to the person who supplied the data.

JSON Lines is not the same format as a pretty-printed JSON document. Each non-empty line must stand on its own as a complete JSON value. In import code, that usually means one JSON object per line.

Decode one record per line

Each line must be valid JSON by itself.

<?php

declare(strict_types=1);

$lines = [
    '{"id":1,"email":"sam@example.com"}',
    '{"id":2,"email":"lee@example.com"}',
];

foreach ($lines as $line) {
    $record = json_decode($line, true, flags: JSON_THROW_ON_ERROR);
    echo $record['id'] . ': ' . $record['email'] . PHP_EOL;
}

// Prints:
// 1: sam@example.com
// 2: lee@example.com

There is no opening [ at the top and no commas between lines. A normal JSON file containing one large array is decoded once. A JSON Lines file is decoded once per record. That difference changes error handling, memory use, progress reporting, and how you design validation.

Most importers should require each line to decode to a JSON object. JSON Lines as a format can contain any JSON value per line, but application imports are usually easier to validate when every line is a record with named fields.

Keep line numbers in errors

Import errors should identify the line that failed. Without a line number, a large import file is painful to debug.

<?php

declare(strict_types=1);

function decodeJsonLine(string $line, int $lineNumber): array
{
    try {
        $record = json_decode($line, true, flags: JSON_THROW_ON_ERROR);
    } catch (JsonException $exception) {
        throw new InvalidArgumentException('Line ' . $lineNumber . ' is not valid JSON.', 0, $exception);
    }

    if (!is_array($record)) {
        throw new InvalidArgumentException('Line ' . $lineNumber . ' must contain a JSON object.');
    }

    return $record;
}

try {
    decodeJsonLine('{"id":', 3);
} catch (InvalidArgumentException $exception) {
    echo $exception->getMessage() . PHP_EOL;
}

// Prints:
// Line 3 is not valid JSON.

This pattern belongs near the import boundary, before records reach application services or database code. Line numbers should refer to the original input, not the count of valid records processed so far. If blank lines are skipped, the next error should still point to the line the user can open in their editor.

Validate every record

Valid JSON is not enough. Each record still needs the fields your importer expects.

<?php

declare(strict_types=1);

function importCustomerRecord(array $record, int $lineNumber): string
{
    if (!isset($record['email']) || !is_string($record['email']) || trim($record['email']) === '') {
        throw new InvalidArgumentException('Line ' . $lineNumber . ' is missing a customer email.');
    }

    return strtolower($record['email']);
}

$record = ['id' => 1, 'email' => 'SAM@EXAMPLE.COM'];

echo importCustomerRecord($record, 1) . PHP_EOL;

// Prints:
// sam@example.com

Line-level validation lets an import report precise failures instead of stopping later with a vague database or type error. It also lets you normalize data at the boundary: trim text, lowercase emails if that is your application rule, convert accepted numeric strings, and reject unknown values before side effects happen.

Do not let every downstream service repeat raw JSON checks. Decode and validate into a smaller internal shape, such as an array with email and sourceId, then pass that shape into the rest of the application.

Stream from a string or file

In production, you normally read from a file handle. For a small example, a generator over a string shows the same control flow.

<?php

declare(strict_types=1);

function linesFromText(string $text): iterable
{
    foreach (explode("\n", $text) as $lineNumber => $line) {
        $line = trim($line);

        if ($line === '') {
            continue;
        }

        yield $lineNumber + 1 => $line;
    }
}

$text = "{\"sku\":\"KB-101\"}\n\n{\"sku\":\"MS-202\"}";

foreach (linesFromText($text) as $lineNumber => $line) {
    $record = json_decode($line, true, flags: JSON_THROW_ON_ERROR);
    echo $lineNumber . ': ' . $record['sku'] . PHP_EOL;
}

// Prints:
// 1: KB-101
// 3: MS-202

Notice that blank lines are skipped but the original line numbers are preserved. That makes error messages match the file the user uploaded.

A real file reader should handle open failures, read failures, encoding expectations, and cleanup. If the reader opens a handle, it should close it even when an exception occurs. Keep the file-reading concern separate from record validation when possible: one function yields numbered lines, another decodes a line, and another validates the decoded record.

Write JSON Lines

When writing JSON Lines, encode each record compactly and append a newline.

<?php

declare(strict_types=1);

$records = [
    ['event' => 'order_created', 'id' => 101],
    ['event' => 'order_paid', 'id' => 101],
];

foreach ($records as $record) {
    echo json_encode($record, JSON_THROW_ON_ERROR) . PHP_EOL;
}

// Prints:
// {"event":"order_created","id":101}
// {"event":"order_paid","id":101}

Do not pretty-print JSON Lines. Pretty JSON spreads one record over multiple lines, which breaks the format. A consumer reading one line at a time would try to decode { as a complete record and fail.

When exporting, decide whether every output record has the same shape and whether large identifiers should be strings. If another system will import your file, document the fields, types, and meaning of missing or null values. JSON Lines makes transport simple, but it does not replace a schema or field contract.

Decide whether to stop or collect errors

Some imports should stop on the first bad line. Others should collect errors and continue so the user can fix a batch of problems at once.

<?php

declare(strict_types=1);

$lines = [
    1 => '{"email":"sam@example.com"}',
    2 => '{"email":""}',
    3 => '{"email":"lee@example.com"}',
];

$emails = [];
$errors = [];

foreach ($lines as $lineNumber => $line) {
    try {
        $record = json_decode($line, true, flags: JSON_THROW_ON_ERROR);

        if (!isset($record['email']) || trim((string) $record['email']) === '') {
            throw new InvalidArgumentException('Line ' . $lineNumber . ' is missing an email.');
        }

        $emails[] = $record['email'];
    } catch (Throwable $exception) {
        $errors[] = $exception->getMessage();
    }
}

echo count($emails) . ' valid records' . PHP_EOL;
echo implode('; ', $errors) . PHP_EOL;

// Prints:
// 2 valid records
// Line 2 is missing an email.

Make this decision deliberately. Silent partial imports are hard to support. If the importer continues after errors, the result should clearly say how many records succeeded, which lines failed, and whether any side effects were committed.

Stop-on-first-error is simpler and often safer for jobs that must be all-or-nothing. Collecting errors is friendlier for user-uploaded files where the user wants a batch of corrections. Both strategies are valid, but the database and transaction design must match the strategy.

Transactions and side effects

Validation and writing are separate phases in many imports. For small files, you can decode and validate every line first, then write only if there are no errors. For large files, validating everything before writing may use too much memory, so you may process in chunks or use a transaction.

If you write records as you go and later lines fail, decide whether previous writes remain. That is a product and data-integrity rule, not just a coding detail. A customer import might reject the whole file on any error. An analytics event import might accept valid lines and report rejected lines. A payment or inventory import needs stricter thinking because partial writes can create real business problems.

Avoid irreversible side effects during loose validation. Sending emails, charging accounts, publishing messages, or deleting data while still discovering line errors can make retries unsafe. Prefer validating first, or use idempotent operations with clear checkpoints.

Operational concerns

JSON Lines files can be large, so importers need limits and observability. Set a maximum upload size, maximum line length, maximum record count, or timeout that fits the use case. A single valid but enormous line can still consume too much memory when decoded.

Progress reporting should count original line numbers, processed records, accepted records, and rejected records separately. Those numbers answer different questions. If a support ticket says line 500 failed, the importer should produce the same line number even if blank lines or comments are skipped.

Logs should include safe metadata such as import ID, file name, line number, error code, and record count. Avoid logging complete raw records by default because imports often contain personal data, tokens, addresses, or business identifiers.

Schema drift and resumable imports

JSON Lines files often come from another system, and that system may change over time. A provider might add fields, rename fields, start sending null, or change an identifier from a number to a string. Decide which changes are acceptable. Many importers ignore unknown fields but reject missing required fields. That policy should be tested so a harmless provider addition does not break an import, while a breaking provider change is caught early.

For long-running imports, think about resumability. If a job fails after 200,000 lines, support staff should not have to guess whether line 199,999 was committed. Store enough progress information to know the last successfully processed line, file checksum, import ID, or external record ID. When retries are possible, make record processing idempotent so rerunning a chunk does not create duplicates.

Duplicate detection is a business rule. Some feeds may legitimately contain multiple events for the same order. Other imports should reject duplicate customer IDs or product SKUs in the same file. If duplicates matter, detect them during validation and include line numbers for both the first and repeated occurrence when that is practical.

Encoding is also part of the contract. JSON text is expected to be Unicode text, and many PHP applications treat it as UTF-8. If files arrive from spreadsheets, old systems, or manual exports, confirm the encoding before decoding and produce an error that explains the import requirement. A generic JSON parse error is less useful when the real problem is the wrong file encoding.

Testing JSON Lines importers

Tests should cover valid lines, blank lines, malformed JSON, valid JSON with the wrong top-level type, missing fields, empty fields, duplicate identifiers if your import forbids them, and line numbers after skipped blank lines. Include one test for the chosen error strategy: stop immediately or collect multiple errors.

For writers, test that each record is one compact line and that the output ends with the newlines expected by your consuming tools. Do not compare pretty-printed JSON when the format is JSON Lines. For large-file behavior, unit tests can cover the generator shape, while integration or performance tests can exercise realistic file sizes.

Review checklist

Before approving JSON Lines code, identify whether the importer streams or materializes the whole file, how line numbers are preserved, where decoding errors are converted into user-facing messages, where record shape is validated, and what happens after the first error. Check that pretty printing is not used for JSON Lines output and that database writes match the chosen error strategy.

What to remember

JSON Lines is for record-by-record processing. Decode one line at a time, preserve original line numbers, validate each record, avoid pretty printing, and choose a clear error strategy before writing to the database. Treat importing as a boundary: it needs limits, useful errors, safe logging, and a deliberate plan for partial success or rollback.