We show that annihilation dynamics between oppositely charged optical vortex pairs can be manipulated by the initial size of the vortex cores, consistent with hydrodynamics. When sufficiently close together, vortices with strongly overlapped cores annihilate more quickly than vortices with smaller cores that must wait for diffraction to cause meaningful core overlap. Numerical simulations and experimental measurements for vortices with hyperbolic tangent cores of various initial sizes show that hydrodynamics governs their motion, and reveal distinct phases of vortex recombination; decreasing the core size of an annihilating pair can prevent the annihilation event.