In this longitudinal prospective observational study performed at a tertiary perinatal referral centre, we aimed to assess maternal distress in pregnancy in women with ultrasound findings of fetal anomaly and compare this with distress in pregnant women with normal ultrasound findings. Pregnant women with a structural fetal anomaly (n = 48) and normal ultrasound (n = 105) were included. We administered self-report questionnaires (General Health Questionnaire-28, Impact of Event Scale-22 [IES], and Edinburgh Postnatal Depression Scale) a few days following ultrasound detection of a fetal anomaly or a normal ultrasound (T1), 3 weeks post-ultrasound (T2), and at 30 (T3) and 36 weeks gestation (T4). Social dysfunction, health perception, and psychological distress (intrusion, avoidance, arousal, anxiety, and depression) were the main outcome measures. The median gestational age at T1 was 20 and 19 weeks in the group with and without fetal anomaly, respectively. In the fetal anomaly group, all psychological distress scores were highest at T1. In the group with a normal scan, distress scores were stable throughout pregnancy. At all assessments, the fetal anomaly group scored significantly higher (especially on depression-related questions) compared to the normal scan group, except on the IES Intrusion and Arousal subscales at T4, although with large individual differences. In conclusion, women with a known fetal anomaly initially had high stress scores, which gradually decreased, resembling those in women with a normal pregnancy. Psychological stress levels were stable and low during the latter half of gestation in women with a normal pregnancy.