Category-selective brain areas exhibit varying levels of neural activity to ipsilaterally presented stimuli. However, in face- and house-selective areas, the neural responses evoked by ipsilateral stimuli in the peripheral visual field remain unclear. In this study, we displayed face and house images using a wide-view visual presentation system while performing functional magnetic resonance imaging (fMRI). The face-selective areas (fusiform face area (FFA) and occipital face area (OFA)) exhibited intense neural responses to ipsilaterally presented images, whereas the house-selective areas (parahippocampal place area (PPA) and transverse occipital sulcus (TOS)) exhibited substantially smaller and even negative neural responses to the ipsilaterally presented images. We also found that the category preferences of the contralateral and ipsilateral neural responses were similar. Interestingly, the face- and house-selective areas exhibited neural responses to ipsilateral images that were smaller than the responses to the contralateral images. Multi-voxel pattern analysis (MVPA) was implemented to evaluate the difference between the contralateral and ipsilateral responses. The classification accuracies were much greater than those expected by chance. The classification accuracies in the FFA were smaller than those in the PPA and TOS. The closer eccentricities elicited greater classification accuracies in the PPA and TOS. We propose that these ipsilateral neural responses might be interpreted by interhemispheric communication through intrahemispheric connectivity of white matter connection and interhemispheric connectivity via the corpus callosum and occipital white matter connection. Furthermore, the PPA and TOS likely have weaker interhemispheric communication than the FFA and OFA, particularly in the peripheral visual field.