Traditional explorations of infectious disease evolution have considered the competition between two cross-reactive strains within the standard framework of disease models. Such techniques predict that diseases should evolve to be highly transmissible, benign to the host and possess a long infectious period: in general, diseases do not conform to this ideal. Here we consider a more holistic approach, suggesting that evolution is a trade-off between adaptive pressures at different scales: within host, between hosts and at the population level. We present a model combining within-host pathogen dynamics and transmission between individuals governed by an explicit contact network, where transmission dynamics between hosts are a function of the interaction between the pathogen and the hosts' immune system, though ultimately constrained by the contacts each infected host possesses. Our results show how each of the scales places constraints on the evolutionary behavior, and that complex dynamics may emerge due to the feedbacks between epidemiological and evolutionary dynamics. In particular, multiple stable states can occur with switching between them stochastically driven.