We study the phase behavior of hard and soft spheres with a fixed dipole moment using Monte Carlo simulations. The spheres interact via a pair potential that is a sum of a hard-core Yukawa (or screened-Coulomb) repulsion and a dipole-dipole interaction. The system can be used to model colloids in an external electric or magnetic field. Two cases are considered: (i) colloids without charge (or dipolar hard spheres) and (ii) colloids with charge (or dipolar soft spheres). The phase diagram of dipolar hard spheres shows fluid, face-centered-cubic (fcc), hexagonal-close-packed (hcp), and body-centered-tetragonal (bct) phases. The phase diagram of dipolar soft spheres shows, in addition to the above mentioned phases, a body-centered-orthorhombic (bco) phase, and is in agreement with the experimental phase diagram [Nature (London) 421, 513 (2003)]. In both cases, the fluid phase is inhomogeneous but we find no evidence of a gas-liquid phase separation. The validity of the dipole approximation is verified by a multipole moment expansion.