Multi-atlas brain segmentation of human brain MR images allows quantification research in structural neuroimaging. To achieve high accuracy and computational efficiency of segmentation relies on a custom subset of atlases for each target subject. However, the criterion for atlas pre-selection remains an open question. In this study, two atlas pre-selection approaches based on location-based feature matching were proposed and compared to random and mutual information-based methods using a database of 47 atlases. A varying number of atlases ranked top with hierarchical structural granularity were compared using Dice overlap. The results indicated that the proposed 4L approach consistently led to the highest level of accuracy at a given number of employed atlases in both adult and geriatric populations. In addition, the proposed two methods (4L and LV) can reduce 20 times computational time compared with the stereotypical mutual information-based method. Our pre-selection strategy would provide better segmentation performance in terms of both accuracy and efficiency. The proposed atlas pre-selection will be further implemented into our online automatic brain image segmentation system (www.mricloud.org).