Send only valid UTF-8 so one bad string can't break queries on a source

[PetrHeinz]

The HTTP device only relabels strings as UTF-8 (force_encoding) without checking them, and it never looks into Arrays or Hash keys. LogEntry also cuts long messages at byte 8,192, which can split a multibyte character in two. Better Stack stores a line with invalid UTF-8 as invalid JSON: its fields read empty, and every json.* query on the whole source then fails with ClickHouse Code 117. In the red-team, a single request with raw bytes in its path, logged by logtail-rack, was enough to break queries on a source.

• Before encoding, every string in the line is made valid UTF-8: the message, nested fields, the context, Array elements and Hash keys.
  • UTF-8, binary (ASCII-8BIT) and US-ASCII strings are read as UTF-8, and invalid bytes become U+FFFD (scrub).
  • Strings in other encodings are converted with encode, so ISO-8859-1 "Montr\xE9al" arrives as Montréal. Characters with no UTF-8 equivalent become U+FFFD.
  • Ruby has no converter for 9 rare encodings, such as Windows-1258 and UTF-7. Strings in those are read as UTF-8 and scrubbed instead of raising.
• LogEntry cuts a message longer than 8,192 bytes on a character boundary, dropping the bytes of the character the cut splits.

Behaviour and compatibility:

• Valid UTF-8 strings are sent byte for byte as before.
• Strings with invalid bytes now arrive as readable text with U+FFFD. Binary strings inside Arrays used to go out as msgpack bin values; they're now text like everywhere else.
• Strings in other encodings now arrive converted instead of mislabelled.
• In a message longer than 8,192 bytes, other invalid bytes are dropped by the same scrub instead of replaced. A long message labelled as binary is still cut at byte 8,192, and a character split there becomes U+FFFD, so it can end up 1 or 2 bytes over the limit.
• logtail-rack needs no change: the strings it logs pass through the same code.

Follow-up after the E2E run of the release candidate (two more commits, the first one only tests and red on CI): force_utf8_encoding duplicated and scrubbed every string, also the ones that are valid UTF-8 already, which are nearly all, and together with #54 encoding a log line got several times slower than on 0.1.20. It now returns a string that is valid UTF-8, or US-ASCII without bytes above 127, as it is, and copies only the others. msgpack sends both the same way, so nothing changes in what arrives. Encoding a typical line, incl. deflate, on Ruby 3.4.9 with 1,000 log lines from the released logtail-rack 0.2.8 (0.1.20: 3.42 µs):

• this PR alone: 7.5 µs before, 5.9 µs after. The rest is copying every Hash and Array, which Encode log entries one at a time so one bad value can't drop a batch #54 replaces with a single pass that only hands strings that aren't valid UTF-8 to force_utf8_encoding;
• merged with Encode log entries one at a time so one bad value can't drop a batch #54: 13.0 µs before, 3.5–3.7 µs after. Encode log entries one at a time so one bad value can't drop a batch #54 has the request latency numbers.

Targets the logtail 0.1.21 patch release. No dependencies on the other open PRs. It pairs with #54, which converts the values msgpack can't encode in the same code path; the two merge cleanly in either order and pass the suite together.

The first commit only adds the tests and is expected to fail on CI; the fix follows in the next commit.

🤖 Generated with Claude Code