How long does it take a quantum particle to return to its origin? As shown previously under repeated projective measurements aimed to detect the return, the closed cycle yields a geometrical phase which shows that the average first detected return time is quantized. For critical sampling times or when parameters of the Hamiltonian are tuned this winding number is modified. These discontinuous transitions exhibit gigantic fluctuations of the return time. While the general formalism of this problem was studied at length, the magnitude of the fluctuations, which is quantitatively essential, remains poorly characterized. Here, we derive explicit expressions for the variance of the return time, for quantum walks in finite Hilbert space. A classification scheme of the diverging variance is presented, for four different physical effects: the Zeno regime, when the overlap of an energy eigenstate and the detected state is small, and when two or three phases of the problem merge. These scenarios present distinct physical effects which can be analyzed with the fluctuations of return times investigated here, leading to a topology-dependent time-energy uncertainty principle.