The magnitude of inbreeding depression, a central parameter in the evolution of plant mating systems, can vary depending on environmental conditions. However, the underlying genetic mechanisms causing environmental fluctuations in inbreeding depression, and the consequences of this variation for the evolution of self-fertilization, have been little studied. Here, we consider temporal fluctuations of the selection coefficient in an explicit genetic model of inbreeding depression. We show that substantial variance in inbreeding depression can be generated at equilibrium by fluctuating selection, although the simulated variance tends to be lower than has been measured in experimental studies. Our simulations also reveal that purging of deleterious mutations does not depend on the variance in their selection coefficient. Finally, an evolutionary analysis shows that, in contrast to previous theoretical approaches, intermediate selfing rates are never evolutionarily stable when the variation in inbreeding depression is due to fluctuations in the selection coefficient on deleterious mutations.