Speech routinely provides cues as to the sex of the talker, in voiced sounds, these cues mainly reflect dimorphism in vocal anatomy. This dimorphism is not symmetrical, however, since during adolescent development, males specifically diverge from a previously shared trajectory with females. We therefore predicted that listeners would show a corresponding perceptual advantage for male sounds in talker-sex discrimination, a hypothesis tested using very brief, one- to eight-cycle vowel segments. The expected performance asymmetry was observed in threshold-like tests of multiple different vowels in Experiments 1-3, and a signal detection design in Experiment 4 helped rule out possible response bias effects. In confirming our counterintuitive prediction, the present study illustrates that a biological and evolutionary perspective can be helpful in understanding indexical cuing in speech.