The sign of the Hall conductivity σ_xy in metals is often used to measure the sign of the charge carriers. In type II superconductors,  simple models of vortex dynamics predict that σ_xy should have the same sign as the one obtained from the normal state charge carriers.  However, Hall measurements in YBCO and other cuprate superconductors showed that σ_xy changes sign as a function of temperature, contradicting these models. Recently it was shown that a sign reversal of σ_xy occurs “naturally” in a generic model for s-wave superconductors, due to a topological transition in the vortex core. We show that a similar effect occurs in a model for d-wave superconductors, and find a rich phase diagram of σ_xy as a function of interaction strength and doping.