Looking Ahead By Looking Back