Unicode Stringize and parsing

[Happypig375]

The current Stringize restricts to ASCII which is easy to edit on a keyboard. However if the output is intended for text display it may be desirable to add a flag to use the full Unicode range. The parser will need to recognize these Unicode outputs and parse unambiguously. For exponentiation, if the superscripts or subscripts can represent the content whole then it is desirable to write ^2 as ² and _2 as ₂, otherwise for nested powers ^ and _ should be kept. However for the final nesting level Unicode superscripts and subscripts are still to be used.

https://unicodemath.org/ may be relevant here. Some nuances like implicit multiplication binding stronger than exponentiation also needs in depth consideration - this may also change normal parsing. We may still want / to be explicitly division (as well as all the presentation focused syntax whereas we have logical nodes), so UnicodeMath parsing and output may be a separate entry point (do we want this at all?) but some ideas still apply to stringize using full unicode range.

[Rafael-SOWNet]

Measured on master at 2dbeedf7. A few of these move the premise, so they are worth having before the design is settled.

1. Stringize is not ASCII-restricted today

The VARIABLE lexer rule already admits Greek, Greek Extended and Cyrillic:

VARIABLE: ('a'..'z'|'A'..'Z'|'Ͱ'..'Ͽ'|'ἀ'..'῿'|'Ѐ'..'ӿ')+ ('_' (…))? ;

and names pass through the printer unchanged, so α ^ 2 + π * ω prints and round-trips as it stands. What is ASCII is the printer's own vocabulary — operators, function names, +oo, NaN. Worth stating that way, because it makes the flag a decision about the vocabulary rather than about the encoding.

2. And that is already the trap this proposal has to avoid: π is a variable

Constants are looked up by name (Variable.ConstantList, keyed pi and e), so the Unicode spellings are free variables:

simplify(sin(pi))  =>  0
simplify(sin(π))   =>  sin(π)     -- π (U+03C0) is a free variable
simplify(ln(e))    =>  1
simplify(ln(е))    =>  ln(е)      -- Cyrillic е (U+0435)
vars(х + x)        =>  х, x       -- two variables, indistinguishable on screen

So a printer that renders pi as π would emit a string that parses back to a different expression, and a valid one — which is exactly how #906 went (NaN printed, read back as a variable). StringizeRoundTripTest states the invariant that has to survive the flag: parsing what it prints gives back the expression it printed, because "anything else makes the printed form a lie, and a silent one".

That makes the ordering fairly clear: whether π/е become the constants or stay free variables has to be decided before anything starts printing them, not after. Either answer is defensible; the homoglyph pairs (Cyrillic е/х, Greek ο) are an argument for deciding it deliberately either way.

3. Superscripts and subscripts are asymmetric, and thinner than they look

Taken from the Unicode compatibility decompositions rather than from memory:

	available	missing
superscript	all digits, + − = ( ), all lowercase Latin	uppercase S X Y Z, and /
subscript	all digits, + − = ( )	lowercase b c d f g q w y z, every uppercase, and /

Two consequences for the "if it can represent the content whole" rule. Rational exponents can never qualify, there being no superscript /, so x ^ (1/2) keeps ^ permanently. And the choice would depend on the characters in the name rather than on the structure — x ^ p can be xᵖ while x ^ q cannot, x_1 can be x₁ while x_A cannot — so two structurally identical expressions print in different notations. Tolerable for a display flag, but it argues for the rule being "superscript for what is always available" (digits and sign) rather than "whenever it happens to fit".

4. A ² survives a normalizer here by luck; a ⁿ does not

Superscripts NFKC-normalize to their base characters, and plenty of things between the printer and the parser normalize. This grammar already reads a digit after a name as a power (Syntax.md: x2 is x ^ 2, (x + 1)2 is (x + 1) ^ 2), so:

x²   --NFKC-->  x2       --parse-->  x ^ 2          correct, by accident
xⁿ   --NFKC-->  xn       --parse-->  variable xn    silently wrong
x⁻¹  --NFKC-->  x−1      --parse-->  token recognition error at '−'

Superscript digits land on their feet purely because that shorthand already exists. Letters and signs do not. A third reason to keep the first version to digits.

5. On "do we want this at all" — there is a precedent on each side

Latexize is already a display-only printer, and it can afford \pi and \infty precisely because nothing parses LaTeX back — there is no reader for it in the library. MathS.Diagnostic.OutputExplicit is already a Setting<bool> that changes printed output ambiently, so an output flag has a shape to copy.

The whole cost is in the second half of the proposal. A display-only Unicode printer is cheap and its risk is bounded by nobody reading it back. Making the parser read it back doubles what StringizeRoundTripTest and SyntaxDocumentedTest have to cover, for every node, and permanently — and the moment the parser accepts π, point 2 stops being a printing question and becomes a change to what existing expressions mean.

6. UnicodeMath as a separate entry point — agreed, with a concrete reason

This grammar's juxtaposition is already asymmetric: 2x is 2 * x, x2 is x ^ 2, x2y is x ^ 2 * y, and x i is x ^ i since i is a number. Adopting UnicodeMath's precedence would therefore reinterpret strings that already parse rather than accept new ones, which is the kind of change that wants its own entry point and its own round-trip test rather than a flag on the shared grammar. MathS.Settings.ExplicitParsingOnly is the safe shape for comparison — it only removes implicit operators, so it cannot change what an already-parsing string means.

Suggested smallest first step

If this is wanted, the version with no round-trip exposure at all:

• a display-only flag, output side only, with the parser untouched;
• superscript digits and sign for integer exponents, subscript digits for indices, ^/_ everywhere else;
• ∞ for +oo and the operator glyphs, since none of them collide with an identifier;
• Greek constants excluded until point 2 is decided;
• gated by running the existing StringizeRoundTripTest corpus a second time with the flag on, asserting the un-flagged reading is still recoverable — so the day the parser side is added, the test that would catch a π regression already exists.

That leaves the parser work and UnicodeMath as separate decisions, which is where the real cost is.

[Happypig375]

We may want to do this as v2.8 before v3 such that v3 can review what we came up with using a holistic view.