awesome-auto-parallelism

Tensor partitioning for heterogeneous deep learning accelerators.

A Double recursive algorithm to search strategies

a deep learning framework that accelerates distributed DNN training by automatically searching for efficient parallelization strategies

PaSE uses a dynamic programming based approach to find an efficient strategy within a reasonable time.

Another paper from Zhihao, Jia. Designed for GNN

Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism

Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Horizontal TP. An intermediate representation and simulator for efficient neural network distribution

a system that uses simple tensor sharding annotations to achieve different parallelism paradigms in a unified way

This code package contains algorithms (proof-of-concept implementation) and input files (profiled DNN models / workloads) from the paper "Piper: Multidimensional Planner for DNN Parallelization" published at NeurIPS 2021. An extension of DNN partitioning

Efficiently training large-scale neural networks with bidirectional pipelines

An Efficient Pipelined Data Parallel Approach for Training Large Model. Succeed from GPipe

published at NeurIPS 2020.

FTPipe can automatically transform sequential implementation into a multi-GPU one.

distributed deep learning with RL based scheduling in heterogeneous environment.

This repository contains the source code implementation of PipeDream and PipeDream-2BW

automatically optimize DNN computation with graph substitution

An A GPipe implementation in PyTorch

A pipeline only system designed for NAS network. Complementary to hybrid parallelism