Biological systems show two structural features on many levels of organization: sparseness, in which only a small fraction of possible interactions between components actually occur; and modularity – the near decomposability of the system into modules with distinct functionality. Here, we studied the origin of modularity and sparseness focusing on the nature of the mutation process, rather than variations in the goal. We use simulations of evolution with different mutation rules. We find that commonly used sum-rule mutations, in which interactions are mutated by adding random numbers, do not lead to modularity or sparseness except for special situations. In contrast, product-rule mutations in which interactions are mutated by multiplying by random numbers– a better model for the effects of biological mutations – lead to sparseness naturally. When the goals of evolution are modular, in the sense that specific groups of inputs affect specific groups of outputs, product-rule mutations lead to modular structure; sum-rule mutations do not. Product-rule mutations generate sparseness and modularity because once they are near a zero element, the effective diffusion rate decreases. Strictly zero terms are fixed-points and near-zero terms remain small under mutations - so that the population becomes concentrated near zero elements. Sum-rule mutations, in contrast, show a constant drift rate regardless of the value of the elements. We obtained a full analytical solution of the dynamics of a simple model by means of Fokker-Planck equations, in excellent agreement with our simulations. While previous works in this field mostly required specific conditions or limited range of parameters for modularity to evolve, the generality and simplicity of the mutational mechanism suggest that it is likely to be broad in its explanatory range.