The waveforms of ultra-short pulses carry, in many cases, important information. Current methods for the temporal characterization of pulses shorter than few femtoseconds have limited sensitivity and impose significant experimental complexity. We propose a new linear and all-optical method inspired by widely-used multi-dimensional phase retrieval algorithms. Our new scheme is based on the spectral measurement of two different unknown sources and their interference. As an example, we focus on the case of spectral polarization measurements of attosecond pulses, relying on their most fundamental property -- being well confined in time. We demonstrate this method numerically reconstructing the temporal profiles of attosecond pulses generated from aligned $CO_2$ molecules. We then demonstrate that the same method can be used to improve lensless imaging methods.