To study the underpinnings of individual differences in subjective well-being (SWB), we tested for associations of SWB with subcortical brain volumes in a dataset of 724 twins and siblings. For significant SWB-brain associations we probed for causal pathways using Mendelian Randomization (MR) and estimated genetic and environmental contributions from twin modeling. Another independent measure of genetic correlation was obtained from linkage disequilibrium (LD) score regression on published genome-wide association summary statistics. Our results indicated associations of SWB with hippocampal volumes but not with volumes of the basal ganglia, thalamus, amygdala, or nucleus accumbens. The SWB-hippocampus relations were nonlinear and characterized by lower SWB in subjects with relatively smaller hippocampal volumes compared to subjects with medium and higher hippocampal volumes. MR provided no evidence for an SWB to hippocampal volume or hippocampal volume to SWB pathway. This was in line with twin modeling and LD-score regression results which indicated non-significant genetic correlations. We conclude that low SWB is associated with smaller hippocampal volume, but that genes are not very important in this relationship. Instead other etiological factors, such as exposure to stress and stress hormones, may exert detrimental effects on SWB and the hippocampus to bring about the observed association.