The PyTorch team announced PyTorch 2.0 at the 2022 PyTorch Conference held yesterday, providing an early version for experience, and said that the stable version will be released in early March 2023.

According to the team, PyTorch 2.0 is their first step towards the next generation of PyTorch 2 series. In the past few years, from PyTorch 1.0 to the latest 1.13, they have innovated and iterated PyTorch, and migrated it to the newly formed PyTorch Foundation, becoming part of the Linux Foundation.

PyTorch 2.0 introduces torch.compile, a compilation mode that speeds up models without changing the model code. Across 163 open-source models spanning vision, NLP, and other domains, the team found training speedups of 38-76% with 2.0.

Second, PyTorch 2.0 is 100% backward compatible: same code base, same API, same way of writing models. The team is calling it 2.0 because of some signature new features, including:

• TorchDynamo can generate FX graphs from bytecode analysis;

• AOTAutograd can generate reverse graphs ahead-of-time;

• PrimTorch introduces a small set of operators to make the backend easier;

• TorchInductor: A DL compiler powered by OpenAI Triton.

PyTorch 2.0 will continue PyTorch’s consistent advantages, including Python integration, imperative style, simple API, etc. Furthermore, PyTorch 2.0 provides the same eager-mode development and user experience while fundamentally changing and enhancing the way PyTorch operates at the compiler level. This version provides faster performance and better support for “Dynamic Shapes” and distributed operations.

In the official blog, the PyTorch team introduced their vision for the 2.0 series:

View documentation for details.