Understanding the contribution of gene-environment interactions (GxE) to complex trait variation can provide insights into mechanisms underlying disease risk, explain sources of heritability, and improve the accuracy of genetic risk prediction. While biobanks that collect genetic and deep phenotypic data over large numbers of individuals offer the promise of obtaining novel insights into GxE, our understanding of the architecture of GxE in complex traits remains limited. We introduce a method that can estimate the proportion of trait variance explained by GxE (GxE heritability) and additive genetic effects (additive heritability) across the genome and within specific genomic annotations. We show that our method is accurate in simulations and computationally efficient for biobank-scale datasets. We applied our method to ≈ 500, 000 common array SNPs (MAF ≥ 1%), fifty quantitative traits, and four environmental variables (smoking, sex, age, and statin usage) measured across ≈ 300, 000 unrelated white British individuals in the UK Biobank. We found 69 trait-environmental variable pairs with significant genome-wide GxE heritability (p < 0.05/200 correcting for the number of trait-E pairs tested) with an average ratio of GxE to additive heritability ≈ 6.8% that include BMI with smoking (ratio of GxE to additive heritability = 6.3 ± 1.1%), WHR (waist-to-hip ratio adjusted for BMI) with sex (ratio = 19.6 ± 2%), LDL cholesterol with age (ratio = 9.8 ± 3.9%), and HbA1c with statin usage (ratio = 11 ± 2%). Analyzing nearly 8 million common and low-frequency imputed SNPs (MAF ≥ 0.1%), we document an increase in genome-wide GxE heritability of about 28% on average over array SNPs. We partitioned GxE heritability across minor allele frequency (MAF) and local linkage disequilibrium values (LD score) of each SNP to observe that analogous to the relationship that has been observed for additive allelic effects, the magnitude of GxE allelic effects tends to increase with decreasing MAF and LD. Testing whether GxE heritability is enriched around genes that are highly expressed in specific tissues, we find significant tissue-specific enrichments that include brain-specific enrichment for BMI and Basal Metabolic Rate in the context of smoking, adipose-specific enrichment for WHR in the context of sex, and cardiovascular tissue-specific enrichment for total cholesterol in the context of age. Our analyses provide detailed insights into the architecture of GxE underlying complex traits.