Skip to content

Qualcomm AI Engine Direct - [GenAI Pipeline] PR5: Model preparation & quantization strategy implementations - #21899

Merged
psiddh merged 1 commit into
pytorch:mainfrom
CodeLinaro:pr4b
Aug 28, 2026
Merged

Qualcomm AI Engine Direct - [GenAI Pipeline] PR5: Model preparation & quantization strategy implementations#21899
psiddh merged 1 commit into
pytorch:mainfrom
CodeLinaro:pr4b

Conversation

@qti-horodnic

Copy link
Copy Markdown
Contributor

Summary

This PR implements the model preparation and quantization strategy implementations, replacing the NotImplementedError stubs with real logic. Each strategy delegates to injectable adapter interfaces (from PR4) for testability.

What's included

Strategy implementations (2 files + 1 __init__ fix):

  • ExecuTorchModelPreparationStrategy: 5-step flow

    • load_modelload_tokenizer (via ModelLoaderAdapter)
    • generate_calibration_data (via separately-injectable CalibrationDataAdapter)
    • Optional tokenizer export for on-device runtime
    • Chat template extraction from tokenizer (with extra_options fallback)
    • Validates input config (model_name, soc_model required)
  • ExecuTorchQuantizationStrategy: Full PT2E single-graph pipeline via QuantizerAdapter

    • export → make_quantizer → prepare_pt2e → calibrate → convert_pt2e
    • Supports quant_dtype, quant_recipe, and per-channel options via extra_options
    • Handles any Iterable as calibration data (lists, DataLoaders, generators)
    • Validates calibration data is non-empty before export
    • Warns (does not fail) when training_data is provided (QAT deferred)
  • strategies/model_preparation/__init__.py: adds missing ExecuTorchModelPreparationStrategy import to __all__

Unit tests:

  • test_executorch_model_preparation_strategy.py
  • test_executorch_quantization_strategy.py
  • test_default_model_preparation_adapter.py
  • test_default_model_preparation_adapter.py

PR Review Checklist

  • All new classes follow single responsibility (one class per file) - Yes.
  • All dependencies are injected via constructor with sensible defaults - Yes.
  • All external calls are behind injectable interfaces - Yes (adapter pattern).
  • Unit tests cover every public method - Yes (100% coverage on strategy impls).
  • Docstrings on all public classes and methods - Yes.
  • Type annotations on all function signatures - Yes.
  • Logging follows the strategy in the LLD - Yes (info on entry/exit, debug per step).

Related PRs

Test plan

Run only tests added in this PR:

python -m pytest \
  backends/qualcomm/genai_pipeline/tests/strategies/model_preparation/ \
  backends/qualcomm/genai_pipeline/tests/strategies/quantization/ \
  -v

Run only this PR's tests with coverage:

python -m pytest \
  backends/qualcomm/genai_pipeline/tests/strategies/model_preparation/ \
  backends/qualcomm/genai_pipeline/tests/strategies/quantization/ \
  --cov=backends/qualcomm/genai_pipeline/strategies/model_preparation \
  --cov=backends/qualcomm/genai_pipeline/strategies/quantization \
  --cov-config=backends/qualcomm/.coveragerc \
  --cov-report=term-missing

Result:

Name                                                                                                     Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------------------------------------------------------------------------------
backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py      68      0     14      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py                        9      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py                61      0     18      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py                            7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py                                9      0      0      0   100%
----------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                      161      0     32      0   100%

Run all genai_pipeline tests:

python -m pytest backends/qualcomm/genai_pipeline/tests/ -v

Run all genai_pipeline tests with coverage:

python -m pytest backends/qualcomm/genai_pipeline/tests/ \
  --cov=backends/qualcomm/genai_pipeline \
  --cov-config=backends/qualcomm/.coveragerc \
  --cov-report=term-missing

Result:

Name                                                                                                     Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------------------------------------------------------------------------------
backends/qualcomm/genai_pipeline/configs/compilation_input_config.py                                        11      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/compilation_output_config.py                                        8      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/inference_input_config.py                                          12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/inference_output_config.py                                          9      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py                                   7      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py                                 12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/quantization_input_config.py                                       13      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/quantization_output_config.py                                       5      0      0      0   100%
backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py                                        5      0      0      0   100%
backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py                               26      0      6      0   100%
backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py                                  14      0      2      0   100%
backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py                                           5      0      0      0   100%
backends/qualcomm/genai_pipeline/engine_proxy.py                                                            20      0      4      0   100%
backends/qualcomm/genai_pipeline/exceptions.py                                                              20      0      6      0   100%
backends/qualcomm/genai_pipeline/genai_pipeline.py                                                          99      7     12      1    93%   190-201
backends/qualcomm/genai_pipeline/pipeline_context.py                                                        52      0     14      0   100%
backends/qualcomm/genai_pipeline/pipeline_stage.py                                                           5      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/compilation_stage.py                                                14      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/inference_stage.py                                                  14      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py                                          14      2      0      0    86%   30, 37
backends/qualcomm/genai_pipeline/stages/quantization_stage.py                                               14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py                              7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py                                 11      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py                   7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py                              14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py                       7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py                                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py      68      0     14      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py                        9      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py                61      0     18      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py                            7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py                                9      0      0      0   100%
----------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                      593      9     76      1    99%

@pytorch-bot

pytorch-bot Bot commented Aug 17, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21899

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 17, 2026
@qti-horodnic

Copy link
Copy Markdown
Contributor Author

@pytorchbot label "release notes: qualcomm"

@pytorch-bot pytorch-bot Bot added the release notes: qualcomm Changes to the Qualcomm backend delegate label Aug 17, 2026
@psiddh

psiddh commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

@claude review this PR also review this
strategies/quantization/default_quantizer_adapter.py — the make_quantizer signature

quant_dtype is a required positional here, but the strategy omits it on the default path expecting export_utils's use_8a8w default — so ExecuTorchQuantizationStrategy() with defaults raises TypeError. Could we give quant_dtype a default here (and create_autospec the adapter in tests)? The bare MagicMock accepts any signature, so this path currently passes CI but would crash in production.

@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown

Claude finished @psiddh's task in 2m 6s —— View job


PR Review: PR5 — Model preparation & quantization strategies

  • Read default_quantizer_adapter.py + export_utils.make_quantizer
  • Trace the default-path call in ExecuTorchQuantizationStrategy
  • Verify test doubles hide the signature mismatch
  • Review the rest of the PR

1. Confirmed: ExecuTorchQuantizationStrategy() with defaults raises TypeError in production 🔴

Your read is exactly right. The chain:

  • default_quantizer_adapter.py:24-30 declares make_quantizer(self, quant_dtype, backend, soc_model, **kwargs)quant_dtype is a required positional, no default.
  • executorch_quantization_strategy.py:149-161 deliberately omits quant_dtype on the default path (comment: "only pass quant_dtype if explicitly provided, so the adapter's default applies otherwise"), calling make_quantizer(backend=..., soc_model=...).
  • But the adapter has no default to fall back to. The use_8a8w default lives one layer deeper, in export_utils.make_quantizer (export_utils.py:653: quant_dtype: Optional[QuantDtype] = QuantDtype.use_8a8w). The wrapper's required positional shadows it.

So the default path raises TypeError: make_quantizer() missing 1 required positional argument: 'quant_dtype' before it ever reaches export_utils.

Why CI is green: _make_mock_adapter() (test file :28-36) uses a bare MagicMock(), which accepts any signature. test_invoke_passes_correct_args_to_make_quantizer (:158-173) even asserts the call is made without quant_dtype — locking in the exact call shape that would crash the real adapter.

Suggested fix — give the wrapper a default that mirrors the intent (let export_utils supply use_8a8w), keeping the lazy-import pattern intact:

def make_quantizer(
    self,
    quant_dtype: Any = None,
    backend: Any = None,
    soc_model: Any = None,
    **kwargs: Any,
) -> Any:
    ...
    make_quantizer_kwargs = {"backend": backend, "soc_model": soc_model_str, **kwargs}
    if quant_dtype is not None:
        make_quantizer_kwargs["quant_dtype"] = quant_dtype
    return _make_quantizer(**make_quantizer_kwargs)

The QuantizerAdapter Protocol (quantizer_adapter.py:34-40) should get the same = None default so the contract matches.

2. Same class of bug, one line over: quant_recipe forwarding 🟠

export_utils.make_quantizer (export_utils.py:652-665) has no quant_recipe parameter — the signature ends at eps=None. But the strategy adds quant_recipe to make_quantizer_kwargs (:158-159) whenever it's set on the config, and DefaultQuantizerAdapter.make_quantizer forwards it straight through **kwargs to _make_quantizer. On the real adapter that's TypeError: make_quantizer() got an unexpected keyword argument 'quant_recipe'.

test_invoke_passes_quant_recipe_from_config (:192-202) passes only because the mock swallows it. Either quant_recipe needs to be consumed/translated inside the adapter (mapped to submodule_qconfig_list or the per-channel flags?), or it shouldn't be forwarded yet. Worth resolving before this reaches production alongside #1.

3. Endorse the create_autospec recommendation ✅

Both bugs above share one root cause: the tests validate against a mock that accepts any signature. Switching _make_mock_adapter() to create_autospec(DefaultQuantizerAdapter, instance=True) would make the mock enforce the real signature, so a defaults-only call and the quant_recipe kwarg both fail in CI instead of production. Recommend applying it to _make_mock_adapter here and, for consistency, the model-preparation strategy tests too.

Fix all three →


Rest of the PR — looks solid

  • ExecuTorchModelPreparationStrategy (executorch_model_preparation_strategy.py): clean 5-step flow, good separation of the calibration-data adapter from the loader. The "tokenizer chat_template wins over extra_options" precedence (:194-200) and the "example inputs come from the model, never the dataset" invariant (:157-166) are well-reasoned and documented. StageError wrapping with re-raise passthrough is consistent with the quant strategy.
  • DefaultModelLoaderAdapter._select_runtime_tokenizer (:199-222): choosing the runtime tokenizer by filename rather than save_pretrained ordering is the right call, and the docstring explains why. artifacts[-1] fallback + warning is a reasonable last resort.
  • calibrate iterating the raw iterable once with torch.no_grad() and leaving generators/DataLoaders untouched (:101-125) matches the config contract nicely.
  • Validation: calibration_data is None rather than a truthiness test (:211-218) correctly avoids consuming a generator's first sample — good catch in the design.

The two production-crash bugs (#1, #2) are the only blocking items; everything else is well-structured and the docstrings/tests are thorough. Since these are review findings I haven't modified any files — happy to apply the fix above if you'd like.
· branch pr4b

@psiddh

psiddh commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Accepting to unblock, @qti-horodnic can you take a look at #1 and possibly #2 (its ok if you want to address in upcoming PRs)

@qti-horodnic

Copy link
Copy Markdown
Contributor Author

Thanks @psiddh, I've addressed the 2 comments. Should be good to merge

@psiddh
psiddh merged commit 72b2536 into pytorch:main Aug 28, 2026
177 of 180 checks passed
@psiddh

psiddh commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Thanks @psiddh, I've addressed the 2 comments. Should be good to merge

Thanks again for the refactor , it is really looking solid

psiddh pushed a commit that referenced this pull request Aug 29, 2026
…ence strategy implementations (#22284)

## Summary

This PR implements the __compilation__ and __inference__ strategy
implementations, completing the strategy layer. Each strategy delegates
to injectable adapter interfaces for testability.

### What's included

#### Strategy implementations:

- `ExecuTorchCompilationStrategy`: Compiles model to .pte artifacts via
`CompilerAdapter`

- Validates model, example_inputs, soc_model, and backend_type are
present
- Passes `example_inputs` explicitly to the adapter (not via
`extra_options`) — mirrors the PR5 fix for the quantization stage
- Delegates to adapter with example_inputs, compile specs, artifact dir,
soc_model, backend_type
  - Filters `context.extra_options` to a compilation-relevant allow-list
  - Returns artifact paths and optional `ETRecord`

- `ExecuTorchInferenceStrategy`: Runs on-device inference via
`DeviceRunnerAdapter`

- Validates artifact_paths and adapter are present (no default adapter —
device config is required)
  - Push → execute → pull results flow
- Two-step protocol: uses `output_data` from execute if present, falls
back to pulled file paths
- Returns inference results, performance metrics, and optional `ETDump`
 
#### `DefaultCompilerAdapter`:

- `compile_model` signature finalised — mirrors
`to_edge_transform_and_lower_to_qnn`
argument-for-argument, so per-graph lowering inputs (`compile_specs`,
`dep_table`,
`passes_job`, `constant_methods`) are explicit parameters rather than
`extra_options` keys
- Body deliberately raises `NotImplementedError`: the version this
package needs is
the multi-graph one (graph-name-keyed dicts, single multi-method `.pte`
for weight
sharing), so it lands with the strategy-level fan-out that calls it
rather than
being written single-graph and then replaced. Inject a custom
`CompilerAdapter` for now.
  
#### Config addition:

- `CompilationInputConfig.example_inputs: Optional[Tuple[Any, ...]]` —
sourced from the model via `ModelLoaderAdapter.get_example_inputs`

#### Orchestrator wiring:

- `genai_pipeline.py` `_run_compilation`: passes
`example_inputs=model_prep_output.example_inputs`

#### Unit tests:

- `test_executorch_compilation_strategy.py` 
- `test_executorch_inference_strategy.py`
- `test_compilation_input_config.py`

### PR Review Checklist

- All new classes follow single responsibility (one class per file) -
Yes.
- All dependencies are injected via constructor with sensible defaults -
Yes.
- All external calls are behind injectable interfaces - Yes (adapter
pattern).
- Unit tests cover every public method - Yes (100% coverage on strategy
impls).
- Docstrings on all public classes and methods - Yes.
- Type annotations on all function signatures - Yes.
- Logging follows the strategy in the LLD - Yes (info on entry/exit,
debug per step).

### Related PRs

- PR 1: Core data model, engine routing & exceptions:
#20409
- PR 2: Strategy interfaces & stage wrappers:
#20795
- PR 3: Pipeline orchestrator:
#21149
- PR 4: Adapter interfaces, default implementations & dataset providers:
#21751
- PR 5: Model preparation & quantization strategies:
#21899
- PR 6: Compilation & inference strategy implementations: this pr.
- PR 7: Integration & E2E tests: pending.

## Test plan

### Run only tests added in this PR:

```
python -m pytest \
  backends/qualcomm/genai_pipeline/tests/strategies/compilation/ \
  backends/qualcomm/genai_pipeline/tests/strategies/inference/ \
  backends/qualcomm/genai_pipeline/tests/configs/test_compilation_input_config.py
  -v
```

### Run only this PR's tests with coverage:

```
python -m pytest \
  backends/qualcomm/genai_pipeline/tests/strategies/compilation/ \
  backends/qualcomm/genai_pipeline/tests/strategies/inference/ \
  backends/qualcomm/genai_pipeline/tests/configs/test_compilation_input_config.py \
  --cov=backends/qualcomm/genai_pipeline/strategies/compilation \
  --cov=backends/qualcomm/genai_pipeline/strategies/inference \
  --cov=backends/qualcomm/genai_pipeline/configs/compilation_input_config \
  --cov-report=term-missing
```

Result:

```
Name                                                                                         Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------------------------------------------------------------------
backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py                     11      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py      45      0     10      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py                  14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py          43      0      6      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py                      7      0      0      0   100%
----------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                          127      0     16      0   100%
```

### Run all `genai_pipeline` tests:

```
python -m pytest backends/qualcomm/genai_pipeline/tests/ -v
```

### Run all tests with coverage:

```
python -m pytest backends/qualcomm/genai_pipeline/tests/ \
  --cov=backends/qualcomm/genai_pipeline \
  --cov-config=backends/qualcomm/.coveragerc \
  --cov-report=term-missing
```

Result:

```
Name                                                                                                     Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------------------------------------------------------------------------------
backends/qualcomm/genai_pipeline/configs/compilation_input_config.py                                        12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/compilation_output_config.py                                        8      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/inference_input_config.py                                          12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/inference_output_config.py                                          9      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py                                   7      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py                                 12      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/quantization_input_config.py                                       13      0      0      0   100%
backends/qualcomm/genai_pipeline/configs/quantization_output_config.py                                       5      0      0      0   100%
backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py                                        5      0      0      0   100%
backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py                               26      0      6      0   100%
backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py                                  14      0      2      0   100%
backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py                                           5      0      0      0   100%
backends/qualcomm/genai_pipeline/engine_proxy.py                                                            20      0      4      0   100%
backends/qualcomm/genai_pipeline/exceptions.py                                                              20      0      6      0   100%
backends/qualcomm/genai_pipeline/genai_pipeline.py                                                          99      7     12      1    93%   190-201
backends/qualcomm/genai_pipeline/pipeline_context.py                                                        52      0     14      0   100%
backends/qualcomm/genai_pipeline/pipeline_stage.py                                                           5      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/compilation_stage.py                                                14      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/inference_stage.py                                                  14      0      0      0   100%
backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py                                          14      2      0      0    86%   30, 37
backends/qualcomm/genai_pipeline/stages/quantization_stage.py                                               14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py                              7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py                                 11      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py                  45      0     10      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py                              14      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py                      43      0      6      0   100%
backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py                                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py      68      0     14      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py                        9      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py                  7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py                61      0     18      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py                            7      0      0      0   100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py                                9      0      0      0   100%
----------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                      668      9     92      1    99%
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: qualcomm Changes to the Qualcomm backend delegate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants