HW-SW Methods for Modeling and Optimizing Communication for Scalable Training of Deep Learning Models