The infamous sign problem leads to an exponential complexity in Monte Carlo simulations of generic many-body quantum systems, but many phases of matter are known to admit a sign-problem-free representative, allowing efficient simulation on classical computers. Motivated by long standing open problems in many-body quantum physics, as well as fundamental questions in quantum complexity, the possibility of incurable sign problems, where a phase of matter admits no sign-problem-free representative, was recently raised but remains largely unexplored. Here we establish the existence, and the geometric origin, of an incurable sign problem in all gapped, chiral, topological phases of matter in 2+1 dimensions. In particular, we exclude the possibility of stoquastic Hamiltonians for bosons, and of sign-problem-free determinantal Monte Carlo algorithms for fermions.