Posterior uncertainty is typically summarized as a credible interval, an interval in the parameter space that contains a fixed proportion - usually 95% - of the posterior's support. For multivariate parameters, credible sets perform the same role. There are of course many potential 95% intervals from which to choose, yet even standard choices are rarely justified in any formal way. In this paper we give a general method, focusing on the loss function that motivates an estimate - the Bayes rule - around which we construct a credible set. The set contains all points which, as estimates, would have minimally-worse expected loss than the Bayes rule: we call this excess expected loss 'regret'. The approach can be used for any model and prior, and we show how it justifies all widely-used choices of credible interval/set. Further examples show how it provides insights into more complex estimation problems.
Keywords: Bayesian Methods; Estimation; Inference.