Impact of nuclear structure on elastic scattering of weakly interacting particles with nuclei