Alloying is widely adopted for tuning the properties of emergent semiconductors for optoelectronic and photovoltaic applications. So far, alloying strategies have primarily focused on engineering bandgaps rather than optimizing charge-carrier transport. Here, we demonstrate that alloying may severely limit charge-carrier transport in the presence of localized charge carriers (e.g., small polarons). By combining reflection-transmission and optical pump-terahertz probe spectroscopy with first-principles calculations, we investigate the interplay between alloying and charge-carrier localization in Cs2AgSbxBi1-xBr6 double perovskite thin films. We show that the charge-carrier transport regime strongly determines the impact of alloying on the transport properties. While initially delocalized charge carriers probe electronic bands formed upon alloying, subsequently self-localized charge carriers probe the energetic landscape more locally, thus turning an alloy's low-energy sites (e.g., Sb sites) into traps, which dramatically deteriorates transport properties. These findings highlight the inherent limitations of alloying strategies and provide design tools for newly emerging and highly efficient semiconductors.