PyTorch Developer Podcast

Edward Yang, Team PyTorch
PyTorch Developer Podcast

The PyTorch Developer Podcast is a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch.

  1. 8月4日

    Compiler collectives

    Compiler collectives are a PT2 feature where by compiler instances across multiple ranks use NCCL collectives to communicate information to other instances. This is used to ensure we consistently decide if inputs or static or dynamic across all ranks. See also PR at https://github.com/pytorch/pytorch/pull/130935

    17分

  2. 4月29日

    TORCH_TRACE and tlparse

    TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.

    15分

  3. 4月21日

    Higher order operators

    Higher order operators are a special form of operators in torch.ops which have relaxed input argument requirements: in particular, they can accept any form of argument, including Python callables. Their name is based off of their most common use case, which is to represent higher order functions like control flow operators. However, they are also used to implement other variants of basic operators and can also be used to smuggle in Python data that is quite unusual. They are implemented using a Python dispatcher.

    17分

  4. 4月12日

    Inductor - Post-grad FX passes

    The post-grad FX passes in Inductor run after AOTAutograd has functionalized and normalized the input program into separate forward/backward graphs. As such, they generally can assume that the graph in question is functionalized, except for some mutations to inputs at the end of the graph. At the end of post-grad passes, there are special passes that reintroduce mutation into the graph before going into the rest of Inductor lowering which is generally aware of passes. The post-grad FX passes are varied but are typically domain specific passes making local changes to specific parts of the graph.

    24分

  5. 3月24日

    CUDA graph trees

    CUDA graph trees are the internal implementation of CUDA graphs used in PT2 when you say mode="reduce-overhead". Their primary innovation is that they allow the reuse of memory across multiple CUDA graphs, as long as they form a tree structure of potential paths you can go down with the CUDA graph. This greatly reduced the memory usage of CUDA graphs in PT2. There are some operational implications to using CUDA graphs which are described in the podcast.

    21分

  6. 3月17日

    Min-cut partitioner

    The min-cut partitioner makes decisions about what to save for backwards when splitting the forward and backwards graph from the joint graph traced by AOTAutograd. Crucially, it doesn't actually do a "split"; instead, it is deciding how much of the joint graph should be used for backwards. I also talk about the backward retracing problem.

    16分

  7. 3月2日

    AOTInductor

    AOTInductor is a feature in PyTorch that lets you export an inference model into a self-contained dynamic library, which can subsequently be loaded and used to run optimized inference. It is aimed primarily at CUDA and CPU inference applications, for situations when your model export once to be exported once while your runtime may still get continuous updates. One of the big underlying organizing principles is a limited ABI which does not include libtorch, which allows these libraries to stay stable over updates to the runtime. There are many export-like use cases you might be interested in using AOTInductor for, and some of the pieces should be useful, but AOTInductor does not necessarily solve them.

    18分

  8. 2月24日

    Tensor subclasses and PT2

    Tensor subclasses allow you to add extend PyTorch with new types of tensors without having to write any C++. They have been used to implement DTensor, FP8, Nested Jagged Tensor and Complex Tensor. Recent work by Brian Hirsh means that we can compile tensor subclasses in PT2, eliminating their overhead. The basic mechanism by which this compilation works is a desugaring process in AOTAutograd. There are some complications involving views, dynamic shapes and tangent metadata mismatch.

    13分
すべて見る（83）

番組について

The PyTorch Developer Podcast is a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch.

情報

  • クリエイター
    Edward Yang, Team PyTorch
  • 配信期間
    21年 - 24年
  • エピソード
    83
  • 制限指定
    不適切な内容を含まない
  • Copyright
    © 2021 - PyTorch Developer Podcast
  • 番組のWebサイト
    PyTorch Developer Podcast

その他のおすすめ

露骨な表現を含むエピソードを聴くには、サインインしてください。

この番組の最新情報をチェック

サインインまたは登録して番組をフォローし、エピソードを保存し、最新のアップデートを入手しましょう。
国または地域を選択

アフリカ、中東、インド

アジア太平洋

ヨーロッパ

ラテンアメリカ、カリブ海地域

米国およびカナダ