Recent experiments in quantum optics provide an ideal platform to study non-equilibrium phase
transitions. In particular, these experiments provided the first realization of a fundamental model
of statistical mechanics, the Dicke model, and demonstrated its second-order phase transition. A
detailed analysis of the experimental observations, however, reveals important discrepancies from the
Dicke model. This is due to decay and dephasing processes, which cannot be taken into account in
the framework of the common mean-field approximation. We include these effects by an appropriate
cumulant expansion of the Lindblad master equations and find significant modifications to the
stability diagram of the system. We find that in addition to the known Dicke phase transition, the
model can undergo a distinctive non-equilibrium transition. This instability is analogous to a lasing
transition, but is induced by counter-rotating terms. This novel effect is demonstrated by recent
experiments on trapped atoms in optical cavities.