Add type hint, update to pyo3 0.27, add automatic type hint generator #1928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

ArthurZucker merged 52 commits into main from more-typehint

Feb 11, 2026

.github/workflows/python.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -129,7 +129,7 @@ jobs: @@
             working-directory: ./bindings/python
             run: |
               source .env/bin/activate
-              ty check py_src tests
+              ty check py_src --exclude py_src/tokenizers/implementations
           - name: Run tests
             working-directory: ./bindings/python
@@ Expand Down @@

bindings/python/Cargo.toml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -14,21 +14,24 @@ serde = { version = "1.0", features = ["rc", "derive"] }
  
    serde_json = "1.0"

    libc = "0.2"

    env_logger = "0.11"

    pyo3 = { version = "0.26", features = ["abi3", "abi3-py39", "py-clone"] }

    pyo3-async-runtimes = { version = "0.26", features = ["tokio-runtime"] }

    pyo3 = { version = "0.27.2",default-features = false, features = ["abi3", "abi3-py39", "py-clone", "experimental-inspect"] }

    pyo3-async-runtimes = { version = "0.27.0", features = ["tokio-runtime"] }

    tokio = { version = "1.47.1", features = ["rt", "rt-multi-thread", "macros", "signal"] }

    once_cell = "1.19.0"

    numpy = "0.26"

    numpy = "0.27"

    ndarray = "0.16"

    itertools = "0.14"

    ahash = { version = "0.8.11", features = ["serde"] }

    pyo3-ffi = "0.27.2"

    [dependencies.tokenizers]

    path = "../../tokenizers"

    [dev-dependencies]

    tempfile = "3.10"

    pyo3 = { version = "0.26", features = ["auto-initialize"] }

    pyo3 = { version = "0.27.2", features = ["auto-initialize"] }

    [features]

    default = ["pyo3/extension-module"]

    default = ["ext-module"]

    ext-module = ["pyo3/extension-module"]

bindings/python/Makefile

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -7,17 +7,18 @@ check_dirs := examples py_src/tokenizers tests
  
    # Format source code automatically

    style:

    	cargo run --manifest-path tools/stub-gen/Cargo.toml

    	python stub.py

    	ruff check  $(check_dirs) --fix 

    	ruff format $(check_dirs)

    	ty check py_src tests

    	ty check py_src --exclude py_src/tokenizers/implementations 

    # Check the source code is formatted correctly

    check-style:

    	python stub.py --check

    	ruff check $(check_dirs)

    	ruff format --check $(check_dirs)

    	ty check py_src tests 

    	ty check py_src --exclude py_src/tokenizers/implementations 

    TESTS_RESOURCES = $(DATA_DIR)/small.txt $(DATA_DIR)/roberta.json

bindings/python/README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -169,6 +169,49 @@ tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")
  
    encoded = tokenizer.encode("I can feel the magic, can you?")

    ```

    ### Typing support and `stub.py`

    ### Typing support and stub generation

    The compiled PyO3 extension does not expose type annotations, so editors and type checkers would otherwise see most objects as `Any`. The `stub.py` helper walks the loaded extension modules, renders `.pyi` stub files (plus minimal forwarding `__init__.py` shims), and formats them so that tools like mypy/pyright can understand the public API. Run `python stub.py` whenever you change the Python-visible surface to keep the generated stubs in sync.

    The compiled PyO3 extension does not expose type annotations, so editors and type checkers would otherwise see most objects as `Any`. To provide full typing support, we use a two-step stub generation process:

    1. **Rust introspection** (`tools/stub-gen/`): Uses `pyo3-introspection` to analyze the compiled extension and generate `.pyi` stub files

    2. **Python enrichment** (`stub.py`): Adds docstrings from the runtime module and generates forwarding `__init__.py` shims

    #### Running stub generation

    The easiest way to regenerate stubs is via `make style`:

    ```bash

    cd bindings/python

    make style

    ```

    This will:

    1. Build the extension with `maturin develop --release`

    2. Run introspection to generate `.pyi` files

    3. Enrich stubs with docstrings via `stub.py`

    4. Format with `ruff`

    #### Running manually

    To run the stub generator directly:

    ```bash

    cd bindings/python

    cargo run --manifest-path tools/stub-gen/Cargo.toml

    python stub.py

    ```

    The stub generator automatically:

    - Builds the extension using maturin

    - Copies the built `.so` to the project root for introspection

    - Detects and sets `PYTHONHOME` for embedded Python (handles uv/venv environments)

    - Generates stubs to `py_src/tokenizers/`

    #### Troubleshooting

    If you encounter Python initialization errors, you can manually set `PYTHONHOME`:

    ```bash

    export PYTHONHOME=$(python3 -c 'import sys; print(sys.base_prefix)')

    cargo run --manifest-path tools/stub-gen/Cargo.toml

    ```

bindings/python/py_src/tokenizers/__init__.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -75,7 +75,7 @@ class SplitDelimiterBehavior(Enum): @@
         CONTIGUOUS = "contiguous"
-    from .tokenizers import (  # type: ignore[import]
+    from .tokenizers import (
         AddedToken,
         Encoding,
         NormalizedString,
@@ Expand Down @@

Navigation Menu

Add type hint, update to pyo3 0.27, add automatic type hint generator #1928

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!