We have successfully imaged the retinal tumor in a mouse model using an ultra-high resolution spectral-domain optical coherence tomography (SD-OCT) designed for small animal retinal imaging. For segmentation of the tumor boundaries and calculation of the tumor volume, we developed a novel segmentation algorithm. The algorithm is based on parametric deformable models (active contours) and is driven by machine learning-based region classification, namely a Conditional Random Field. With this algorithm we are able to obtain the tumor boundaries automatically, while the user can specify additional constraints (points on the boundary) to correct the segmentation result, if needed. The system and algorithm were successfully applied to studies on retinal tumor progression and monitoring treatment effects quantitatively in a mouse model of retinoblastoma.