Few studies have examined the association between environmental phthalate exposure and children's neurocognitive development. This longitudinal study examined cognitive function in relation to pre-and postnatal phthalate exposure in children 2-12 years old. We recruited 430 pregnant women in their third trimester in Taichung, Taiwan from 2001-2002. A total of 110, 79, 76, and 73 children were followed up at ages 2, 5, 8, and 11, respectively. We evaluated the children's cognitive function at four different time points using the Bayley and Wechsler tests for assessing neurocognitive functions and intelligence (IQ). Urine samples were collected from mothers during pregnancy and from children at each follow-up visit. They were analyzed for seven metabolite concentrations of widely used phthalate esters. These esters included monomethyl phthalate, monoethyl phthalate, mono-butyl phthalate, mono-benzyl phthalate, and three metabolites of di(2-ethylhexyl) phthalate, namely, mono-2-ethylhexyl phthalate, mono(2-ethyl-5-hydroxyhexyl) phthalate, and mono(2-ethyl-5-oxohexyl) phthalate. We constructed a linear mixed model to examine the relationships between the phthalate metabolite concentrations and the Bayley and IQ scores. We found significant inverse associations between the children's levels of urinary mono(2-ethyl-5-oxohexyl) phthalate and the sum of the three metabolites of di(2-ethylhexyl) phthalate and their IQ scores (β = -1.818; 95% CI: -3.061, -0.574, p = 0.004 for mono(2-ethyl-5-oxohexyl) phthalate; β = -1.575; 95% CI: -3.037, -0.113, p = 0.035 for the sum of the three metabolites) after controlling for maternal phthalate levels and potential confounders. We did not observe significant associations between maternal phthalate exposure and the children's IQ scores. Children's but not prenatal phthalate exposure was associated with decreased cognitive development in the young children. Large-scale prospective cohort studies are needed to confirm these findings in the future.