Treat primitive type names as predefined identifiers instead of keywords

[pepone]

Primitive type names such as bool, int32, and string do not need to be lexer keywords. They can instead be ordinary identifiers bound to predefined types in the global type namespace.

With this model:

module Example

struct string {}

struct Value {
    local: string       // Example::string
    primitive: ::string // Global primitive
}

A module can also share a name with a primitive because modules and types occupy different namespaces:

module int32

struct Value {
    field: int32 // Global primitive
}

Proposed changes

• Lex primitive names as identifiers rather than dedicated keyword tokens.
• Predefine primitive types in the AST’s global type namespace.
• Resolve primitive and user-defined type references through the same scoped lookup.
• Allow user-defined types to shadow primitives normally.
• Use the requested symbol category during lookup so that modules and types with the same name can coexist.
• Continue accepting escaped spellings such as \string for compatibility, but make them equivalent to string.

This should remove the primitive grammar production, immediate primitive patching, and dedicated primitive lookup. Result, Sequence, and Dictionary can remain grammar keywords because they introduce constructed-type syntax.

[InsertCreativityHere]

I'm pretty sure these points are already true today on main and 0.4.x:

• Predefine primitive types in the AST’s global type namespace
• Allow user-defined types to shadow primitives normally
• Resolve primitive and user-defined type references through the same scoped lookup (although my PR undoes this one)

So the only real changes would be:

• Lex primitive names as identifiers rather than dedicated keyword tokens.
• Use the requested symbol category during lookup so that modules and types with the same name can coexist.
• Continue accepting escaped spellings such as \string for compatibility, but make them equivalent to string

[InsertCreativityHere]

I think the core idea of treating primitives as normal identifiers is fine. It's just a matter of framing:
A) We treat them as special keywords to highlight their special primitive nature
B) We treat them as regular identifiers which are defined at global scope

These are both reasonable. Option A is probably what most languages do, but we're allowed to do our own thing!
And B would let us remove the grammar rule and tokens for primitives. Not huge, but a simplification still.

But I don't see what problem this addresses, and for me, this example seems dubious:

module int32

struct Value {
    field: int32 // Global primitive
}

1. Right now we have a single lookup table. And if I do ast.lookup_table["int32"] it's ambiguous if this should give
   the module, or the primitive type. This is the whole reason for Fix Ambiguity in Elements using Escaped Primitive Keywords #813 removing the primitives from this table.
   From Use the requested symbol category during lookup ... it sounds like you're suggesting we have 2 lookup tables?
   But I don't see how 2 tables is better than having 2 functions (find_primitive_node vs find_node_by_id) like my PR.

2. I would actually expect the opposite answer to field: int32 there. This line is saying more than it seems:
   A module can also share a name with a primitive because modules and types occupy different namespaces
   Right now this isn't a true statement - they both exist in a single global psuedo-scope. We could change this at a the
   concept level, but at the implementation level, I don't see how to do this without breaking the idea of
   Resolve primitive and user-defined type references through the same scoped lookup.

3. This is technically a breaking change. Something like this already works perfectly right now:

module Example
struct \int32 { f: int32 }

This is a regular struct, with a single primitive int32 field, which will compile okay. This proposal would change the meaning of int32 to point to the struct itself, turning this into an illegal self-referential struct. That's not inherently wrong, but changing the meaning like this is probably not something to do in a patch release.

TL;DR

I'm fine with the idea of switching primitives to be treated like identifiers, although I also don't really see any
inherent benefit to doing this.

But mostly, I don't think it's possible to do both of these things at the same time:
Resolve primitive and user-defined type references through the same scoped lookup. and
Use the requested symbol category during lookup so that modules and types with the same name can coexist..
Either we have 2 different ways of looking things up, or "int32" will always be ambiguous in the lookup table itself.

[bernardnormier]

The current proposal does not explain well the benefits of this switch. I believe this proposal is largely inspired by Rust's built-in primitive types: they are not keywords but they are nevertheless special and known to rustc.

The benefits are:

1. consistent look-up / shadowing rules for these built-in primitive types and user-defined types
2. the ability for users to define a custom type (typically) that is later replaced by new built-in primitive type ... when this happens, no need to escape the custom type--it just naturally shadows the new built-in primitive type

For example: I define my own custom int128:

module BigInts

custom int128

then in the next version of slicec, int128 is a built-in primitive type. I can keep using my BigInts::int128 without any change to my Slice definitions. In within module BigInts, int128 still means BigInts::int128. No escaping and other change required.

On the other hand, doing:

module int32
struct string {}

are not good use-cases, on the contrary.

predefined types in the global type namespace.

I'd rather put these built-in primitive types in a named module, such as PrimitiveTypes (consistent with WellKnownTypes), for example PrimitiveTypes::string.

Putting built-in primitive types in the global "::" module is problematic, as currently either it does not exist or (logically) only top-level modules are allowed in this global namespace.

[InsertCreativityHere]

I think there's really 2 separate issues tied up in this one proposal:
A) Removing the keyword aspect of the primitive types, so they just become regular identifiers.
B) Altering our lookup logic to address the current bugs + the effects of point 1

---

Sounds like we all agree to do part A: I'll remove the special keyword-ness of our primitive types.
I think the benefits are vastly overstated though.

The benefits are:

1) consistent look-up / shadowing rules for these built-in primitive types and user-defined types
2) the ability for users to define a custom type (typically) that is later replaced by new built-in primitive type ... when this happens, no need to escape the custom type--it just naturally shadows the new built-in primitive type

Point 1 is not a benefit of this proposal. Users can already shadow the primitive types just fine, and I don't see this proposal changing that.
All that would change is you could write int32 now instead of needing to write \int32. But the shadowing is unchanged. And the look-up won't be purely consistent under it either. There will still be some "special"ness in looking up these types, since they're implicitly in every scope.

Point 2 is true, but just absolutely incredibly niche. Adding new primitive keywords is ultra rare. There's a serious chance that we will never do it. Multiply that by the odds of a user already having a type with the exact same name? This feels negligible to me.

[InsertCreativityHere]

Well, actually there's still one point to settle... How to disambiguate the primitive types.

@pepone suggests putting in the global scope ::int32.
To do this, we'd need to add a parser rule that rejects modules named after primitive types. Otherwise the identifier ::int32 is ambiguous.

@bernardnormier suggests putting them in a dedicated module named PrimitiveTypes.
This would restrict users defining types in a module named PrimitiveTypes.

---

I don't know which I lean towards more. Having them at global scope will make the lookup easier. We'd only need one entry in the AST's lookup table (::int32). Logically placing them in a module would mean we'd need two entries: ::int32 and ::PrimitiveTypes::int32. But the module approach makes them feel more like a regular type instead of a special construct.

Implementation wise, I guess we'd have an issue with module int32 with both of the approaches here. But, I mean, that's just absolutely pathological to do. So I don't think it's a real concern.