Behavioral flexibility requires directing feedforward sensory information to appropriate targets. In the superior colliculus, divergent outputs orchestrate different responses to visual threats, but the circuit organization enabling the flexible routing of sensory information remains unknown. To determine this structure, we focused on inhibitory projection (Gad2) neurons. Trans-synaptic tracing and neuronal recordings revealed that Gad2 neurons projecting to the lateral geniculate nucleus (LGN) and the parabigeminal nucleus (PBG) form two separate populations, each receiving a different set of non-retinal inputs. Inhibiting the LGN- or PBG-projecting Gad2 neurons resulted in opposing effects on behavior; increasing freezing or escape probability to visual looming, respectively. Optogenetic activation of selected inputs to the LGN- and PBG-projecting Gad2 cells predictably regulated responses to visual threat. These data suggest that projection-specific sampling of brain-wide inputs provides a circuit design principle that enables visual inputs to be selectively routed to produce context-specific behavior.