Phyllodes tumors are uncommon breast neoplasms that constitute 1-2% of breast malignancies. Invasive ductal carcinoma in the epithelial component of phyllodes tumor is very rare. When carcinoma is detected within the specimen, the management of treatment changes completely. We report a rare case of invasive ductal carcinoma arising in a giant borderline malignancy phyllodes tumor in a 51-year-old female patient. A painful 20 cm mass was found in her right breast, and a needle biopsy revealed fibroadenoma or benign phyllodes tumor, and a total mastectomy was performed. Pathological results showed that a borderline malignant phyllodes tumor coexisted with invasive ductal carcinoma. We explained that axillary surgery was necessary because invasive cancer was diagnosed after surgery, but the patient requested follow-up using images. Endocrine therapy was performed as postoperative adjuvant therapy, and the follow-up is underway without recurrence.