Humans learn the rules that govern how the elements of their language are organized over an input that is often not homogeneous (it might contain noise, or even include rules from different linguistic systems, as it might be the case for bilinguals). In the present study we explore the conditions under which participants can learn an abstract rule when it is presented in a heterogeneous context. Results from six experiments show that listeners can learn a token-independent rule even if it is presented together with some exemplars that implement a different regularity (Experiment 1a and 1b). In fact, learning rules from an input containing several patterns does not seem to differ from learning them from an input containing only one (Experiment 1c). More surprisingly, we observed that listeners can even learn an abstract rule that is only implemented over 10% of the exemplars that compose a familiarization stream (Experiments 2a and 2b). When the proportion of tokens implementing the target and the non-target rules is balanced, we did not observe any learning (Experiment 3). Our results suggest that listeners use differences in relative frequency to keep separate linguistic rules apart. This allows them to learn different abstract regularities from a non-homogeneous linguistic signal.