The gut microbiota, as an important factor affecting host health, plays a significant role in the occurrence and development of diabetes mellitus (DM), and the mechanism may be related to excessive endotoxins, altered short-chain fatty acids (SCFAs), and disordered bile acid metabolism. Traditional Chinese medicine (TCM) has a long history of treating DM, but its mechanism is not very clear. Recent research has suggested that Chinese herbal medicine can improve glucose metabolism by remodeling the gut microbiota, which opens new avenues for further research on hypoglycemic mechanisms. This review presents the recent progress of Chinese herbs, herbal extracts, and herbal compound preparations in treating DM through regulating the gut microbiota and summarizes the main mechanisms involved, namely, anti-inflammatory and antioxidative effects, protecting the intestinal barrier and inhibiting lipotoxicity. In addition, some suggestions for improvement are also proposed.