Rat liver 3alpha-hydroxysteroid dehydrogenase (3alpha-HSD), a member of the aldoketoreductase superfamily, inactivates circulating steroid hormones using NAD(P)H as cofactor. Despite determination of the 3alpha-HSD.NADP+ binary complex structure, the functional elements that dictate the binding of steroids remain unclear (Bennett, M.J., Schlegel, B.P., Jez, J.M., Penning, T.M., and Lewis, M. (1996) Biochemistry 35, 10702-10711). Two tryptophans (Trp86 and Trp227) near the active site may have roles in substrate binding, and their fluorescence may be quenched upon binding of NADPH. Trp86 is located within an apolar cleft, while Trp227 is found on an opposing loop near the active site. A third tryptophan, Trp148, is on the periphery of the structure. To investigate the roles of these tryptophans in protein fluorescence and ligand binding, we generated three mutant enzymes (W86Y, W148Y, and W227Y) by site-directed mutagenesis. Spectroscopic measurements on these proteins showed that Trp148 contributed the most to the enzyme fluorescence spectra, with Trp227 adding the least. Trp86 was identified as the tryptophan quenched by bound NADPH through an energy transfer mechanism. The W86Y mutant altered binding of cofactor (a 3-fold increase in Kd for NADPH) and steroid (a 7-fold increase in Kd for testosterone). This mutation also dramatically decreased the catalytic efficiency observed with one-, two-, and three-ring substrates and decreased the binding affinity for nonsteroidal anti-inflammatory drugs but had little effect on the binding of aldose reductase inhibitors. Interestingly, mutation of Trp227 significantly impaired steroid binding (a 22-fold increase in Kd for testosterone), but did not alter binding of cofactor, smaller substrates, or inhibitors. Kinetically, the W148Y mutant was similar to wild-type enzyme. Our results demonstrate that Trp86 and the apolar cleft is part of the substrate binding pocket. In addition, we propose a role for Trp227 and its associated loop in binding steroids, but not small substrates or inhibitors, most likely through interaction with the C- and D-rings of the steroid. This work provides the first evidence that tryptophans on opposite sides of the apolar cleft are part of the steroid binding pocket and suggests how the enzyme may discriminate between nonsteroidal anti-inflammatory drugs and aldose reductase inhibitors like zopolrestat. A model of how androstanedione binds in the apolar cleft is developed. These data provide further evidence that loop structures in members of the aldoketoreductase superfamily are critical determinants of ligand binding.