PyTorch Distributed Overview

学完你能

• 在 PyTorch 官方教程文档的 Distributed 分组中定位分布式训练相关教程的入口
• 按官方列出的顺序逐条查看 DDP、FSDP2、张量并行、流水线并行、RPC 等教程的标题与链接（各教程具体内容见各教程标题，本页节选未提供）
• 结合自身需要，从这些标题中挑出下一步要打开的教程（选型依据本页节选未提供）

适合你吗

课程特点

内容侧重

按内容目录判断

练习与项目

本页为概览与索引页，后续练习请以所打开教程页面的说明为准；节选未说明练习所需环境。

学习时长

17 个课节原站未给出统一学习时长；实际用时取决于已有基础。

内容费用

免费PyTorch 官方文档教程页，节选可直接阅读导航与教程条目，未见付费墙或登录限制。

内容覆盖

• PyTorch Distributed Overview
• Distributed Data Parallel in PyTorch - Video Tutorials
• Getting Started with Distributed Data Parallel
• Writing Distributed Applications with PyTorch
• Getting Started with Fully Sharded Data Parallel (FSDP2)
• Introduction to Libuv TCPStore Backend
• Large Scale Transformer model training with Tensor Parallel (TP)
• Introduction to Distributed Pipeline Parallelism
• Customize Process Group Backends Using Cpp Extensions
• Getting Started with Distributed RPC Framework
• Implementing a Parameter Server Using Distributed RPC Framework
• Implementing Batch RPC Processing Using Asynchronous Executions
• Interactive Distributed Applications with Monarch
• Debugging Hangs with Flight Recorder Using TorchComms and Debug Server
• Combining Distributed DataParallel with Distributed RPC Framework
• Distributed Training with Uneven Inputs Using the Join Context Manager
• Distributed training at scale with PyTorch and Ray Train