Modeling Deep Context in Spatial and Temporal Domain