Placing a sinusoidally driven air capacitor in the intracavity beam path of a mode-locked few-cycle Ti:sapphire oscillator, we measure the influence of the electro-optical Kerr effect on the carrier-envelope phase of the laser pulses. Using a capacitor length of only 8 cm at atmospheric pressure, we observe a Kerr-induced frequency modulation of the carrier-envelope beat note. From the measured frequency excursion, we determine a Kerr constant of the order of 10-27 m2/V2, which is found to agree with theoretically computed hyperpolarizabilities of the nitrogen and oxygen molecules. The carrier-envelope phase only depends on the dispersion of the hyperpolarizability, which has been previously found very challenging to measure in the gas phase. Our substantially more sensitive measurement method for the electro-optic Kerr effect in air may prove a valuable tool for non-contact measurements of high voltages in power grids and even for monitoring atmospheric electric fields during thunderstorms.