fix orphaned utf-16 high surrogate dropping the next code unit

[soma0212]

StreamInUtf16 handles a high surrogate that isn't followed by a low surrogate by queuing a replacement character, but when the following unit is an ordinary BMP code unit it then queues ch (the orphaned high surrogate) instead of chLow (the unit that actually followed). That encodes a lone surrogate as ill-formed UTF-8 (U+D800 becomes ED A0 80) and drops the real character. It only shows up on malformed UTF-16 input, so the parser silently corrupts the scalar instead of substituting U+FFFD the way the replacement path intends.

Queue chLow so the orphan collapses to a single U+FFFD and the following character is still decoded. The bug lives in the decoder because that is where the replacement-character contract is enforced; callers only see the already-decoded UTF-8. Added a regression test next to the existing encoding tests covering a high surrogate followed by a plain character.

[SGSSGene]

Thank you for this PR!

Never mind: \xEF\xBF\xBD encodes the replacement character in UTF-8

--- Old message:
I need some help understanding the expected result (I am not well versed in Utf16):

Why do we expect "\xEF\xBF\xBD" "A"? Or the question phrased slightly differently: why dos 0xD8000 translate to \xEF\xBF\xBD?
I see that \xEF\xBF is a BMP, but I do not know what \xEF\xBF stands for (this does not seem right).
And why does it get followed by a \xBD?

It seems like I am missing something important!

[soma0212]

Right, EF BF BD is U+FFFD (the replacement character) in UTF-8, so the orphaned high surrogate collapses to a single U+FFFD and the A that used to get dropped comes through as-is. Happy to expand the test comment if that mapping is worth spelling out for the next reader.

[SGSSGene]

Right, EF BF BD is U+FFFD (the replacement character) in UTF-8, so the orphaned high surrogate collapses to a single U+FFFD and the A that used to get dropped comes through as-is. Happy to expand the test comment if that mapping is worth spelling out for the next reader.

That is a good I idea! That might help the next person who reads it!