Factorization guided lightweight neural networks for visual analysis