Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/_data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,10 @@ lnav:
url: /tutorials/accelerator-abstraction-interface/
- title: 'Accelerator Setup Guides'
url: /tutorials/accelerator-setup-guide/
- title: 'Automatic Tensor Parallelism'
- title: 'Automatic Tensor Parallelism (Inference)'
url: /tutorials/automatic-tensor-parallelism/
- title: 'Automatic Tensor Parallelism (Training)'
url: /tutorials/autotp-training/
- title: 'Autotuning'
url: /tutorials/autotuning/
- title: 'BingBertSQuAD Fine-tuning'
Expand Down
2 changes: 1 addition & 1 deletion docs/_tutorials/automatic-tensor-parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Automatic Tensor Parallelism for HuggingFace Models"
tags: inference
---

> **Note:** This tutorial covers AutoTP for **inference**. For **training** with tensor parallelism and ZeRO optimization, see [AutoTP Training API](autotp-training).
> **Note:** This tutorial covers AutoTP for **inference**. For **training** with tensor parallelism and ZeRO optimization, see [Automatic Tensor Parallelism (Training)](/tutorials/autotp-training/).

# Contents
* [Introduction](#introduction)
Expand Down
14 changes: 6 additions & 8 deletions docs/_tutorials/autotp-training.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
---
title: "AutoTP Training API"
title: "Automatic Tensor Parallelism (Training)"
tags: training tensor-parallelism
---

# AutoTP Training API

This tutorial covers the **AutoTP Training API** for combining tensor parallelism with ZeRO optimization during training. For inference-only tensor parallelism, see [Automatic Tensor Parallelism for HuggingFace Models](automatic-tensor-parallelism).
This tutorial covers **Automatic Tensor Parallelism** for combining tensor parallelism with ZeRO optimization during training. For inference-only tensor parallelism, see [Automatic Tensor Parallelism (Inference)](/tutorials/automatic-tensor-parallelism/).

## Contents
- [Introduction](#introduction)
Expand Down Expand Up @@ -87,7 +85,7 @@ If your model matches a built-in preset, set `tensor_parallel.preset_model` in t
}
```

For the list of available presets, see [supported models](../code-docs/training#autotp-supported-models).
For the list of available presets, see [supported models](/code-docs/training#autotp-supported-models).



Expand Down Expand Up @@ -183,6 +181,6 @@ For Grouped Query Attention with different Q/K/V sizes:

## See Also

- [Automatic Tensor Parallelism for Inference](automatic-tensor-parallelism)
- [ZeRO Optimization](zero)
- [DeepSpeed Configuration](config-json)
- [Automatic Tensor Parallelism (Inference)](/tutorials/automatic-tensor-parallelism/)
- [ZeRO Optimization](/tutorials/zero/)
- [DeepSpeed Configuration](/docs/config-json/)