Shubhankar (Shuby) Gahlot

The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).

Distributed training in deep learning (DL) is common practice as data and models grow. The current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale, and model accuracy deterioration with an increase in global batch size. Present solutions focus on improving message exchange efficiency as well as implementing techniques to tweak batch sizes and models in the training process. The loss of training accuracy typically happens because the loss function gets trapped in a local minima. We observe that the loss landscape minimization is shaped by both the model and training data and propose a data optimization approach that utilizes machine learning to implicitly smooth out the loss landscape resulting in fewer local minima. Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.

Knowledge graphs are graphical representations of knowledge using entities and their relationships. Properly structured, they can be a powerful way to surface latent relationships among well- defined entities. By breaking down sentences into their semantic components, an Earth science corpus can be represented as a graph, with verbs acting as the relationship edges between entity nodes. This allows domain information to be captured with high precision. However, since there are multiple verbs in English that can be used to denote the same meaning, this high precision comes at the cost of sparsity of connections and query results. In this presentation, we show a technique where we disambiguate the meaning of a verb in a given sentence using word2vec, with the aim to consolidate it into one among a limited number of synonym sets. This leads to a denser graph and more matches for a given query.

We present a labeled machine learning (ML) training dataset derived from Sentinel 1 C-band synthetic aperture radar (SAR) data for flood events. In this paper, we detail the steps to collect, pre-process, label, curate, and catalog the training dataset. Development of benchmark ML models and usage of the training datasets for a data science competition are also presented.