In order to develop an effective preventive intervention for depression, this cross-sectional study aimed to analyze the relationship between past lifestyle, social activity, and depressive symptoms among community-dwelling elderly people in Japan (n = 887). The Geriatric Depression Scale was used for assessing depressive symptoms. The lifestyle of middle-aged people was assessed using a recall report. The prevalence of depressive symptoms among the subjects was 12.9%. Logistic regression analyses showed that physical exercise, daily intake of well-balanced meals and milk products, and the presence of hobbies were related to the absence of depressive symptoms among both men and women. Among men, avoidance of excess salt and fat was related to the absence of depressive symptoms, while among women, moderate alcohol consumption and good relationships with neighbors were related. Physical exercise, dietary education, and promotion of social activities among middle-aged people were recommended for prevention of depressive symptoms.