Much of the fossil record for dogs consists of mandibles. However, can fossil canid mandibles be reliably identified as dogs or wolves? 3D geometric morphometric analysis correctly classifies 99.5% of the modern dog and wolf mandibles. However, only 4 of 26 Ust'-Polui fossil mandibles, a Russian Arctic site occupied from 250BCE to 150CE, were identified as dogs and none of the 20 Ivolgin mandibles, an Iron Age site in southern Russia, were identified as dogs. Three of the Ust'-Polui mandibles and 8 of the Ivolgin mandibles were identified as wolves. In contrast, all 12 Ivolgin skulls and 5 Ust'-Polui skulls were clearly identified as dogs. Only the classification of the UP6571 skull as a dog (Dog Posterior Probability = 1.0) was not supported by the typical probability. Other evidence indicates these canids were domesticated: they were located within human dwellings, remains at both sites have butchery marks indicating that they were consumed, and isotope analysis of canid and human remains from Ust'-Polui demonstrate that both were consuming freshwater protein; indicating that the humans were feeding the canids. Our results demonstrate that the mandible may not evolve as rapidly as the cranium and the mandible is not reliable for identifying early dog fossils.