We propose and analyze a new way for obtaining an adiabatic geometric phase for light, via the sum-frequency-generation nonlinear process. The state of light is represented by the complex amplitudes at two different optical frequencies, coupled by the second order nonlinearity of the medium. The dynamics of this system is then shown to be equivalent to that of a spin-1/2 particle in a magnetic field, which in turn can be rotated adiabatically on the Bloch sphere. When the input wave itself is an eigenstate of the magnetic field equivalent, the geometric phase is manifested as a pure phase factor. Two adiabatic rotation schemes, based on specific modulations of the quasi-phase-matching poling parameters, are discussed. In the first, the geometric phase is shown to be sensitive to the pump intensity variations, as a result of the Bloch sphere deformation; The second can be utilized for the realization of nonlinear-optics-based geometric phase plates. Moreover, non-closed adiabatic trajectories are investigated, which are expected to provide a robust and broadband geometric wavefront shaping in the sum frequency.