Although clinical observations suggest that alexithymic individuals have a deficit in their ability to recognize emotional stimuli and that this deficit is not simply due to a problem in verbal labeling, these two hypotheses have not been empirically confirmed. Three hundred eighty participants in a community survey without current or past histories of psychiatric disorder completed two independent measures of alexithymia [the Levels of Emotional Awareness Scale (LEAS) and the Toronto Alexithymia Scale (TAS-20)] and the Perception of Affect Task (PAT), a 140-item measure of the ability to match emotion stimuli. The PAT includes four subtasks that require the subject to match verbal or nonverbal emotion stimuli with verbal or nonverbal emotion responses. The subtasks include matching sentences and words (verbal-verbal), faces and words (nonverbal-verbal), sentences and faces (verbal-nonverbal), and faces and photographs of scenes (nonverbal-nonverbal). Across the entire sample, higher (alexithymic) TAS-20 and lower LEAS scores were both correlated with lower accuracy rates on each of the subtasks of the PAT (p < .001), accounting for 10.5% and 18.4% of the variance, respectively. Fifty-one subjects met TAS-20 criteria for alexithymia. Alexithymic individuals scored lower than other subjects on purely nonverbal matching, purely verbal matching, and mixed verbal-nonverbal matching (all p < .001). These results suggest that alexithymia is associated with impaired verbal and nonverbal recognition of emotion stimuli and that the hallmark of alexithymia, a difficulty in putting emotion into words, may be a marker of a more general impairment in the capacity for emotion information processing.