We study spin susceptibility and magnetic order at the edges/surfaces of two- and three-dimensional topological
insulators when the Fermi surface is nested. We find that, due to spin-momentum locking as well as time-reversal
symmetry, spin susceptibility at the nesting wave vector has a strong helical feature. It follows then that a
helical spin density wave state emerges at low temperature due to Fermi surface nesting. The helical feature
of spin susceptibility also has a profound impact on the magnetic order in magnetically doped surface of
three-dimensional topological insulators. In such a system, from the mean-field Zener theory, we predict a helical
magnetic ordered state.