Sexual transmission and persistence of Zika virus (ZIKV) in the testes pose new challenges for controlling virus outbreaks and developing live-attenuated vaccines. It has been shown that testicular infection of ZIKV is initiated in the testicular interstitium, followed by spread of the virus in the seminiferous tubules. This leads to testicular damage and/or viral dissemination into the epididymis and eventually into semen. However, it remains unknown which cell types are targeted by ZIKV in the testicular interstitium, and what is the specific order of infectious events leading to ZIKV invasion of the seminiferous tubules. Here, we demonstrate that interstitial leukocytes expressing mir-511-3p microRNA are the initial targets of ZIKV in the testes, and infection of mir-511-3p-expressing cells in the testicular interstitium is necessary for downstream infection of the seminiferous tubules. Mir-511-3p is expressed concurrently with CD206, a marker of lineage 2 (M2) macrophages and monocyte derived dendritic cells (moDCs). Selective restriction of ZIKV infection of CD206-expressing M2 macrophages/moDCs results in the attenuation of macrophage-associated inflammatory responses in vivo and prevents the disruption of the Sertoli cell barrier in vitro. Finally, we show that targeting of viral genome for mir-511-3p significantly attenuates early ZIKV replication not only in the testes, but also in many peripheral organs, including spleen, epididymis, and pancreas. This incriminates M2 macrophages/moDCs as important targets for visceral ZIKV replication following hematogenous dissemination of the virus from the site of infection.