Emotions have been shown to modulate memory processes. However, the neuronal substrate underlying these modulatory effects is largely unknown. Using event-related functional magnetic resonance imaging (fMRI), we investigated whether the context of emotional encoding modulates brain activation predictive for subsequent recall of emotionally neutral material. While inferior frontal activation predicted recall in general, our data show that in a positive encoding context, recall was predicted by activation of right anterior parahippocampal and extrastriate visual brain areas, whereas in a negative encoding context, recall was predicted by activation of the amygdala. Thus, we could demonstrate that successful episodic encoding is differentially modulated by emotional context. These results contribute to the understanding of the interaction of emotion and cognition and moreover are of general relevance for studies of episodic memory.