Radiologic orbital imaging provides important information in the diagnosis and management of orbital inflammation. However, the diagnostic value of orbital imaging is not well elucidated. This study aimed to investigate the diagnostic accuracy of orbital imaging to diagnose orbital inflammatory diseases and its ability to detect active inflammation. We collected 75 scans of 52 patients (49 computed tomography (CT) scans; 26 magnetic resonance (MR) imaging scans). Clinical diagnoses included thyroid eye disease (TED) (41 scans, 31 patients), non-specific orbital inflammation (NSOI) (22 scans, 14 patients), sarcoidosis (4 scans, 3 patients), IgG4-related ophthalmic disease (IgG4-ROD) (5 scans, 3 patients), and granulomatosis with polyangiitis (GPA) (3 scans, 1 patient). Two experienced neuroradiologists interpreted the scans, offered a most likely diagnosis, and assessed the activity of inflammation, blinded to clinical findings. The accuracy rate of radiological diagnosis compared to each clinical diagnosis was evaluated. Sensitivity and specificity in detecting active inflammation were analyzed for TED and NSOI. The accuracy rate of radiologic diagnosis was 80.0% for IgG4-ROD, 77.3% for NSOI, and 73.2% for TED. Orbital imaging could not diagnose sarcoidosis. Orbital CT had a sensitivity of 50.0% and a specificity of 75.0% to predict active TED using clinical assessment as the gold standard. The sensitivity/specificity of orbital MR was 83.3/16.7% for the detection of active NSOI. In conclusion, orbital imaging is accurate for the diagnosis of IgG4, NSOI, and TED. Further studies with a large number of cases are needed to confirm this finding, especially with regard to uncommon diseases. Orbital CT showed moderate sensitivity and good specificity for identifying active TED.