The relaxation of the structure in the electrical double layer at the ionic liquid|gold interface to the steps of the electrode potential has been studied using surface plasmon resonance (SPR) measurements. The relaxation of the SPR resonance angle occurs on the order of 100 s, which is distinctively slower than the RC time constant of the cell, about 0.1 s. A relaxation model considering the distribution of the time constants well reproduces the SPR curves. The potential steps to the positive direction lead to two orders of magnitude slower relaxation of the SPR resonance angle than those to the negative direction. The ultraslow relaxation and its asymmetry to the potential-step directions are likely to be caused by slow inter-layer dynamics of ions in the ionic multilayers and interaction of ions with gold, respectively.