Understanding the complexity and sustainability of infrastructure development is crucial for reconciling economic growth, human well-being, and environmental conservation. However, previous studies on infrastructure's material metabolism were mainly conducted on a global or national scale, due largely to lack of more spatially refined data, and thus could not reveal the spatial patterns and dynamics on a city scale. Here, we integrated material flow analysis (MFA) and geographical information system (GIS) data to uncover the spatiotemporal patterns of the material stocks and flows accompanying the infrastructure development at a high spatial resolution for the case of Shanghai, China. From 1980 to 2010, material stocks and waste output flows of Shanghai's infrastructure system exhibited a significant increase from 83 to 561 million metric tons (Mt) and from 2 to 17 Mt, respectively. Input flows peaked in 2005 because of the economic slowdown and stepped-up policies to cool the housing market. Spatially, the center and peri-urban areas were the largest container of material stocks and biggest generator of demolition waste, while suburban areas absorbed 58%-76% of material inputs. Plans to make the city more compact will enhance the service capacity of stocks but may also increase the use of more energy and emissions-intensive construction materials (e.g., steel). Prolonging the service lifetime of infrastructure through proper management and increasing the recycling and reuse rate of demolition waste are also identified as highly efficient strategies.