The ability to image at high speeds is necessary for biological imaging to capture fast-moving or transient events or to efficiently image large samples. However, due to the lack of rigidity of biological specimens, carrying out fast, high-resolution volumetric imaging without moving and agitating the sample has been a challenging problem. Pupil-matched remote focusing has been promising for high NA imaging systems with their low aberrations and wavelength independence, making it suitable for multicolor imaging. However, owing to the incoherent and unpolarized nature of the fluorescence signal, manipulating this emission light through remote focusing is challenging. Therefore, remote focusing has been primarily limited to the illumination arm, using polarized laser light to facilitate coupling in and out of the remote focusing optics. Here, we introduce a novel optical design that can de-scan the axial focus movement in the detection arm of a microscope. Our method splits the fluorescence signal into S and P-polarized light, lets them pass through the remote focusing module separately, and combines them with the camera. This allows us to use only one focusing element to perform aberration-free, multi-color, volumetric imaging without (a) compromising the fluorescent signal and (b) needing to perform sample/detection-objective translation. We demonstrate the capabilities of this scheme by acquiring fast dual-color 4D (3D space + time) image stacks with an axial range of 70 μm and camera-limited acquisition speed. Owing to its general nature, we believe this technique will find its application in many other microscopy techniques that currently use an adjustable Z-stage to carry out volumetric imaging, such as confocal, 2-photon, and light sheet variants.