Psychophysical experiments reveal a counterintuitive effect, the “contrast paradox”, in the Vernier hyperacuity task (SB Stevenson, LK Cormack, ​Vision Res​, 2000). In this task a subject reports on the relative position of two misaligned vertical bars. Increasing the contrast of both bars improves performance. However, increasing the contrast of only one bar degrades performance - even though naively more information about the position of the higher-contrast bar is arriving at the retina. Here we argue that the contrast paradox may arise due to the continuous, random movement of the eye during fixation, known as fixational drift. Motion of the eyes might shift the apparent position of the bars, relative to their instantaneous location. If the upper and lower bars have the same contrast, their apparent shifts are identical. On the other hand, if the upper and lower bars differ in contrast, their apparent shifts may differ. A bias may arise in their perceived separation, depending on the detailed trajectory of the eye. This trajectory-dependent bias may contribute to an increased error rate in the task. To quantitatively examine this hypothesis, we consider a biologically plausible decoding strategy, a quadratic decoder, which exploits pairs of nearly synchronous spikes to perform the task. We train the quadratic decoder on simulated spike trains, generated by a model retina in response to the vernier stimulus, amid fixational drift. We find that a simple, feed-forward model retina which includes spatio-temporal filtering by ganglion cell receptive fields is insufficient to explain the contrast paradox. However, a previously reported component of retinal processing (DW Crevier, M Meister, ​J Neurophysiol​, 1998), the `contrast-gain control', generates a trajectory and contrast dependent shift in the apparent location of the bars. The contrast paradox is reproduced by the quadratic decoder when taking this nonlinear effect into account