The insulin-like growth factor type 1 receptor (IGF1R) is involved in progression of breast cancer and resistance to systemic treatment. Targeting IGF1R signaling may, therefore, be beneficial in systemic treatment. We report the effect of IGF1R expression on prognosis in invasive ductal breast carcinoma (IDC), the most common type of breast cancer. Immunohistochemistry was performed on tumor tissue of a consecutive cohort of 429 female patients treated for operable primary IDC. Associations between IGF1R expression with clinicopathological parameters, disease free survival (DFS) and breast cancer specific survival (BCSS) were evaluated by multivariate analyses focusing on ER-positive and triple negative IDC (TN-IDC). To enlarge the TN-IDCs cohort, we analyzed a combined dataset of 51 TN-IDC tumors from our series with 64 TN-IDCs with similar clinicopathological parameters. Patients with tumors expressing cytoplasmic IGF1R have a longer DFS and BCSS (DFS: HR 0.46, 95% CI 0.27-0.49, P = 0.005, BCSS: HR 0.38, 95% CI 0.19-0.74, P = 0.005). This effect was most prominent in ER-positive tumors. However, in a combined series of 105 TN-IDCs cytoplasmic IGF1R expression was associated with a shorter DFS (HR = 2.29, 95% CI 1.08-4.84, P = 0.03), also when combined in a multivariate model, including well-known prognostic factors (HR 2.06; 95% CI 0.95-4.47; P = 0.07). IGF1R expression in ER-positive IDC is strongly related to a favorable DFS and BCSS, but to a shorter DFS in TN-IDC tumors. This divergent effect of IGF1R expression in subgroups of IDC may affect selection of patients for IGF1R targeted therapy.