People have particular difficulty ignoring distractors that depict faces. This phenomenon has been attributed to the high level of biological significance that faces carry. The current study aimed to elucidate the mechanism by which faces gain processing priority. We used a focused attention paradigm that tracks the influence of a distractor over time and provides a measure of inhibitory processing. Upright famous faces served as test stimuli and inverted versions of the faces as well as upright non-face objects served as control stimuli. The results revealed that although all of the stimuli elicited similar levels of distraction, only inverted distractor faces and non-face objects elicited inhibitory effects. The lack of inhibitory effects for upright famous faces provides novel evidence that reduced inhibitory processing underlies the mandatory nature of face processing.