The mechanism for phonon scattering by nanostructures and by point defects in nanostructured silicon (Si) and the silicon germanium (Ge) alloy and their thermoelectric properties are investigated. We found that the thermal conductivity is reduced by a factor of 10 in nanostructured Si in comparison with bulk crystalline Si. However, nanosize interfaces are not as effective as point defects in scattering phonons with wavelengths shorter than 1 nm. We further found that a 5 at. % Ge replacing Si is very efficient in scattering phonons shorter than 1 nm, resulting in a further thermal conductivity reduction by a factor of 2, thereby leading to a thermoelectric figure of merit 0.95 for Si95Ge5, similar to that of large grained Si80Ge20 alloys.