Non-blocking Smile parser silently drops truncated root-level value at end-of-input

[cowtowncoder]

When input ends in the middle of a root-level value, the Smile non-blocking parser (NonBlockingByteArrayParser) does not report an error: after endOfInput(), nextToken() returns null and the parser closes cleanly, so the truncated value is silently dropped.

The blocking SmileParser reports JsonEOFException for the same input, and the non-blocking parser itself does report EOF correctly when the truncated value is inside an Array or Object.

Reproduction

SmileFactory f = new SmileFactory();
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
try (JsonGenerator g = f.createGenerator(bytes)) {
    g.writeBinary(new byte[300]); // same with a long String
}
byte[] doc = Arrays.copyOf(bytes.toByteArray(), bytes.size() - 5); // truncate

try (JsonParser p = f.createNonBlockingByteArrayParser()) {
    ByteArrayFeeder feeder = (ByteArrayFeeder) p.getNonBlockingInputFeeder();
    feeder.feedInput(doc, 0, doc.length);
    p.nextToken();      // NOT_AVAILABLE
    feeder.endOfInput();
    p.nextToken();      // null; p.isClosed() == true -- no exception
}

Observed (2.18 branch):

Case	Result
Async, root-level value truncated (binary or String)	nextToken() returns null, no exception
Async, same value truncated inside [ ... ]	JsonEOFException: Unexpected end-of-input: expected close marker for Array
Blocking, root-level value truncated	JsonEOFException: Unexpected end-of-input for Binary value (7-bit): expected 300 payload bytes ...

Cause

NonBlockingParserBase._eofAsNextToken() only reports an error for open Array/Object scopes. At root level, a partially-decoded token (current token NOT_AVAILABLE, mid-value minor state) is not considered.

Suggested fix

In nextToken()'s _endOfInput branch, report invalid EOF when the current token is NOT_AVAILABLE (i.e. a token was in progress), similar to what jackson-core's non-blocking JSON parser does in _finishTokenWithEOF().

Found while reviewing #826.

[cowtowncoder]

Note for whoever fixes this: there is an interaction with a change being made in #826. Watch for it, so the fix doesn't report a false truncation.

#826 adds a skip state to NonBlockingByteArrayParser (MINOR_VALUE_SKIP_7BIT_BODY). When a BigInteger/BigDecimal fails maxNumberLength validation, the parser throws, and on the next nextToken() it skips the value's encoded content so that parsing can continue with the following token.

Planned follow-up: when that skip ends exactly at the end of the current input chunk, currentToken() briefly reports null instead of NOT_AVAILABLE, until more input is fed. The natural fix is to stay in the skip state (with zero bytes left) and return NOT_AVAILABLE until the next feed:

if (_pending64 > 0 || _inputPtr >= _inputEnd) {
    return _updateTokenToNA();
}

Interaction with this issue: in that zero-bytes-left state the parser is technically "mid-token" (_currToken == NOT_AVAILABLE), even though the value has been fully consumed. If this issue is fixed as suggested above (report invalid EOF whenever the current token is NOT_AVAILABLE at endOfInput()), a document that ends right after a skipped oversized root-level number would be wrongly reported as truncated.

So the EOF check here should treat "MINOR_VALUE_SKIP_7BIT_BODY with nothing left to skip" as a complete value, not a truncated one. Alternatively, the follow-up could be done in a way that doesn't leave the parser mid-token.