E-cadherin (CDH1) is a master regulator of epithelial cell adherence junctions and a well-established tumor suppressor in Invasive Lobular Carcinoma (ILC). Intriguingly, somatic inactivation of E-cadherin alone in mouse mammary epithelial cells (MMECs) is insufficient to induce tumor formation. Here we show that E-cadherin loss induces extrusion of luminal MMECs to the basal lamina. Remarkably, E-cadherin-deficient MMECs can breach the basal lamina but do not disseminate into the surrounding fat pad. Basal lamina components laminin and collagen IV supported adhesion and survival of E-cadherin-deficient MMECs while collagen I, the principle component of the mammary stromal micro-environment did not. We uncovered that relaxation of actomyosin contractility mediates adhesion and survival of E-cadherin-deficient MMECs on collagen I, thereby allowing ILC development. Together, these findings unmask the direct consequences of E-cadherin inactivation in the mammary gland and identify aberrant actomyosin contractility as a critical barrier to ILC formation.