Learning to Understand People via Local, Global and Temporal Reasoning