Remove from My Interests
In this session, participants will get a taste of state-of-the-art techniques for scaling Deep Learning on GPU clusters. We present SuperML, a general and efficient communication layer for machine learning, which can scale neural network training to hundreds of GPU nodes. SuperML builds on three main ideas: decentralization, which allows algorithms to converge without a centralized coordinator (parameter server) or all-to-all communication, communication quantization, which significantly speeds up point-to-point messaging, and structured sparsity, by which SuperML induces model updates which only have a limited number of non-zero entries. From the technical perspective, SuperML provides a new implementation of the classic MPI standard, re-designed and re-implemented to provide efficient support for quantization and sparsity. We illustrate the performance characteristics of SuperML on CSCS Piz Daint, Europe's most powerful supercomputer, and on Amazon EC2, improving upon other highly optimized implementations such as CrayMPI and NVIDIA NCCL.
Do Not Sell My Personal Information