No
Yes
View More
View Less
Working...
Close
OK
Cancel
Confirm
System Message
Delete
My Schedule
An unknown error has occurred and your request could not be completed. Please contact support.
Scheduled
Scheduled
Wait Listed
Personal Calendar
Speaking
Conference Event
Meeting
Interest
There aren't any available sessions at this time.
Conflict Found
This session is already scheduled at another time. Would you like to...
Loading...
Please enter a maximum of {0} characters.
{0} remaining of {1} character maximum.
Please enter a maximum of {0} words.
{0} remaining of {1} word maximum.
must be 50 characters or less.
must be 40 characters or less.
Session Summary
We were unable to load the map image.
This has not yet been assigned to a map.
Search Catalog
Reply
Replies ()
Search
New Post
Microblog
Microblog Thread
Post Reply
Post
Your session timed out.
This web page is not optimized for viewing on a mobile device.Visit this site in a desktop browser or download the mobile app to access the full set of features.
2018 GTC San Jose

S8668 - Scaling Machine Learning through Decentralization, Quantization, and Structured Sparsity

Session Speakers
Session Description

In this session, participants will get a taste of state-of-the-art techniques for scaling Deep Learning on GPU clusters. We present SuperML, a general and efficient communication layer for machine learning, which can scale neural network training to hundreds of GPU nodes. SuperML builds on three main ideas: decentralization, which allows algorithms to converge without a centralized coordinator (parameter server) or all-to-all communication, communication quantization, which significantly speeds up point-to-point messaging, and structured sparsity, by which SuperML induces model updates which only have a limited number of non-zero entries. From the technical perspective, SuperML provides a new implementation of the classic MPI standard, re-designed and re-implemented to provide efficient support for quantization and sparsity. We illustrate the performance characteristics of SuperML on CSCS Piz Daint, Europe's most powerful supercomputer, and on Amazon EC2, improving upon other highly optimized implementations such as CrayMPI and NVIDIA NCCL.


Additional Information
AI/DL Research, Accelerated Analytics, Supercomputing
Cloud Services, Government / National Labs, Higher Education / Research
All technical
Talk
50 minutes
Session Schedule