Efficient Algorithms for Device Placement of DNN Graph Operators
Explore efficient algorithms for optimal device placement of deep neural network graph operators, enhancing throughput and minimizing latency. Dynamic programming and integer programming approaches are detailed, outperforming human experts and baselines across modern DNN workloads.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Efficient Algorithms for Device Placement of DNN Graph Operators Jakub Tarnawski Amar Phanishayee Nikhil Devanur Divya Mahajan Fanny Nina Paravecino Microsoft Amazon NeurIPS 2020
Zillion Zillion- -dollar question: how to train DNNs efficiently? dollar question: how to train DNNs efficiently? Data parallelism: replicate model train on disjoint samples But: communication (weight sync) very expensive SOTA models are huge and can t fit on one worker Figures: courtesy of PipeDream
Model parallelism Model parallelism For high worker utilization, use pipelining (schedules proposed by PipeDream or GPipe) Sample Sample Sample Sample Sample 1 2 3 4 5 How to split the DNN? Figures: courtesy of PipeDream
How to split the DNN? We isolate the structured combinatorial optimization problem at the core of device placement for both training and inference Our contributions Our contributions We give algorithms to solve it optimally
Our contributions Our contributions 1. Dynamic Programming approach to maximize throughput Deals with DNN operator/layer graphs that are arbitrary DAGs Highly efficient Finds non-trivial optimal splits Outperforms human experts and baselines on 7 modern DNN workloads 2. Integer Programming approach Can find non-contiguous splits 3. Integer Programming approach To minimize single-sample latency (for inference) Thank you!
