Immune checkpoint inhibitors (ICIs) are a remarkable advancement in cancer therapeutics; however, a substantial proportion of patients develop severe immune-related adverse events (irAEs). Understanding and predicting irAEs is a key to advancing precision immuno-oncology. Immune checkpoint inhibitor-mediated colitis (IMC) is a significant complication from ICI and can have life-threatening consequences. Based on clinical presentation, IMC mimics inflammatory bowel disease, however the link is poorly understood. We hypothesized that genetic susceptibility to Crohn's disease (CD) and ulcerative colitis (UC) may predispose to IMC. We developed and validated polygenic risk scores for CD (PRSCD) and UC (PRSUC) in cancer-free individuals and assessed the role of each of these PRSs on IMC in a cohort of 1,316 patients with non-small cell lung cancer who received ICIs. Prevalence of all-grade IMC in our cohort was 4% (55 cases), and for severe IMC, 2.5% (32 cases). The PRSUC predicted the development of all-grade IMC (HR=1.34 per standard deviation [SD], 95% CI=1.02-1.76, P=0.04) and severe IMC (HR=1.62 per SD, 95% CI=1.12-2.35, P=0.01). PRSCD was not associated with IMC or severe IMC. The association between PRSUC and IMC (all-grade and severe) was consistent in an independent pan-cancer cohort of patients treated with ICIs. Furthermore, PRSUC predicted severe IMC among patients treated with combination ICIs (OR = 2.20 per SD, 95% CI = 1.07-4.53, P=0.03). This is the first study to demonstrate the potential clinical utility of a PRS for ulcerative colitis in identifying patients receiving ICI at high risk of developing IMC, where risk reduction and close monitoring strategies could help improve overall patient outcomes.