Split out of the discussion on #1242, because it is true today and independent of whether Unicode printing is ever added.
The VARIABLE lexer rule admits Greek, Greek Extended and Cyrillic alongside Latin:
VARIABLE: ('a'..'z'|'A'..'Z'|'Ͱ'..'Ͽ'|'ἀ'..''|'Ѐ'..'ӿ')+ ('_' (…))? ;
Named constants are matched by name (Variable.ConstantList, keyed pi and e). The two facts together produce three traps, measured on 2dbeedf7:
simplify(sin(pi)) => 0 simplify(sin(π)) => sin(π) -- π is U+03C0, a free variable
simplify(ln(e)) => 1 simplify(ln(е)) => ln(е) -- е is U+0435, Cyrillic
vars(х + x) => х, x -- х is U+0445; two variables
π is not pi. Typing or pasting the character every source of mathematics uses for the constant gives a free variable, and the expression simply does not simplify. No error, no warning — sin(π) is a perfectly good expression about an unknown.
- Cyrillic
е is not e. Same shape, and this one is not even a deliberate choice by the person typing it: е and e are indistinguishable in every font.
х + x has two variables and prints as х + x. The printed form round-trips correctly — that is not the problem. The problem is that a human reading it, or comparing two expressions by eye, cannot tell them apart, and Solve on one of them silently treats the other as a parameter.
None of this is a round-trip defect (StringizeRoundTripTest is satisfied throughout) and none of it is a parser bug in the strict sense — every case follows the documented rules exactly. It is a case where the rules compose into something surprising.
Why raise it separately from #1242
#1242 asks whether the printer should emit Unicode. That proposal can only be answered after deciding what the parser should do with π, because a printer emitting π for the constant would produce a string that reads back as a free variable — but the reverse is not true. The reading question stands on its own and is worth settling whether or not anything ever prints Unicode.
Options, none of which is obviously right
- Alias the spellings.
π → the pi constant, е/ℯ → e. Cheap, and it is what most systems do. It costs anyone currently using π as a free variable, and it does nothing for х vs x.
- Diagnose confusables. Warn (or refuse under a setting) when one expression contains two identifiers that are confusable under UTS #39 — which catches
х + x and, as a side effect, sin(π) if π is aliased. There is no diagnostic channel for parsing today, which is the real cost.
- Narrow the identifier rule. Drop the Cyrillic and Greek Extended ranges. Cheapest, and the most likely to break someone.
- Nothing, and document it.
Syntax.md currently does not mention that non-Latin identifiers are accepted at all, so even this option is a change.
I do not have a recommendation. The question I would want answered first is why the Cyrillic range is in the grammar — if it is there for Russian-language variable names then aliasing Greek and leaving Cyrillic is inconsistent, and if it arrived by copying a character-class it may simply be removable.
Measured on 2dbeedf7; every line above is reproducible from a fresh parse.
Split out of the discussion on #1242, because it is true today and independent of whether Unicode printing is ever added.
The
VARIABLElexer rule admits Greek, Greek Extended and Cyrillic alongside Latin:Named constants are matched by name (
Variable.ConstantList, keyedpiande). The two facts together produce three traps, measured on2dbeedf7:πis notpi. Typing or pasting the character every source of mathematics uses for the constant gives a free variable, and the expression simply does not simplify. No error, no warning —sin(π)is a perfectly good expression about an unknown.еis note. Same shape, and this one is not even a deliberate choice by the person typing it:еandeare indistinguishable in every font.х + xhas two variables and prints asх + x. The printed form round-trips correctly — that is not the problem. The problem is that a human reading it, or comparing two expressions by eye, cannot tell them apart, andSolveon one of them silently treats the other as a parameter.None of this is a round-trip defect (
StringizeRoundTripTestis satisfied throughout) and none of it is a parser bug in the strict sense — every case follows the documented rules exactly. It is a case where the rules compose into something surprising.Why raise it separately from #1242
#1242 asks whether the printer should emit Unicode. That proposal can only be answered after deciding what the parser should do with
π, because a printer emittingπfor the constant would produce a string that reads back as a free variable — but the reverse is not true. The reading question stands on its own and is worth settling whether or not anything ever prints Unicode.Options, none of which is obviously right
π→ thepiconstant,е/ℯ→e. Cheap, and it is what most systems do. It costs anyone currently usingπas a free variable, and it does nothing forхvsx.х + xand, as a side effect,sin(π)ifπis aliased. There is no diagnostic channel for parsing today, which is the real cost.Syntax.mdcurrently does not mention that non-Latin identifiers are accepted at all, so even this option is a change.I do not have a recommendation. The question I would want answered first is why the Cyrillic range is in the grammar — if it is there for Russian-language variable names then aliasing Greek and leaving Cyrillic is inconsistent, and if it arrived by copying a character-class it may simply be removable.
Measured on
2dbeedf7; every line above is reproducible from a fresh parse.