Both HIV infection and high levels of early life stress (ELS) have been related to abnormalities in frontal-subcortical structures, yet the combined effects of HIV and ELS on brain structure and function have not been previously investigated. In this study we assessed 49 non-demented HIV-seropositive (HIV+) and 47 age-matched HIV-seronegative healthy control (HC) adults. Levels of ELS exposure were quantified and used to define four HIV-ELS groups: HC Low-ELS (N = 20); HC High-ELS (N = 27); HIV+ Low-ELS (N = 24); HIV+ High-ELS (N = 25). An automated segmentation tool measured volumes of brain structures known to show HIV-related or ELS-related effects; a brief neurocognitive battery was administered. A significant HIV-ELS interaction was observed for amygdala volumes, which was driven by enlargements in HIV+ High-ELS participants. The HIV+ High-ELS group also demonstrated significant reductions in psychomotor/processing speed compared with HC Low-ELS. Regression analyses in the HIV+ group revealed that amygdala enlargements were associated with higher ELS, lower nadir CD4 counts, and reduced psychomotor/processing speed. Our results suggest that HIV infection and high ELS interact to increase amygdala volume, which is associated with neurocognitive dysfunction in HIV+ patients. These findings highlight the lasting neuropathological influence of ELS and suggest that high ELS may be a significant risk factor for neurocognitive impairment in HIV-infected individuals.