Electrostatic charging affects the many-body spectrum of Andreev states, yet its influence on their microwave properties has not been elucidated. We developed a circuit quantum electrodynamics probe that, in addition to transition spectroscopy, measures the microwave susceptibility of different states of a semiconductor nanowire weak link with a single dominant (spin-degenerate) Andreev level. We found that the microwave susceptibility does not exhibit a particle-hole symmetry, which we qualitatively explain as an influence of Coulomb interaction. Moreover, our state-selective measurement reveals a large, π-phase shifted contribution to the response common to all many-body states which can be interpreted as arising from a phase-dependent continuum in the superconducting density of states.