Trellis for efficient data and task management in the VA Million Veteran Program

Paul Billing Ross; Jina Song; Philip S Tsao; Cuiping Pan

doi:10.1038/s41598-021-02569-5

Trellis for efficient data and task management in the VA Million Veteran Program

Sci Rep. 2021 Dec 1;11(1):23229. doi: 10.1038/s41598-021-02569-5.

Authors

Paul Billing Ross^{1

2}, Jina Song^{1

2}, Philip S Tsao^{3

4}, Cuiping Pan⁵

Affiliations

¹ Stanford Center for Genomics and Personalized Medicine, Stanford University, Stanford, CA, USA.
² Palo Alto Epidemiology Research and Information Center for Genomics, VA Palo Alto, CA, USA.
³ Department of Medicine, Stanford University, Stanford, CA, USA. pstao@stanford.edu.
⁴ Palo Alto Epidemiology Research and Information Center for Genomics, VA Palo Alto, CA, USA. pstao@stanford.edu.
⁵ Palo Alto Epidemiology Research and Information Center for Genomics, VA Palo Alto, CA, USA. cuiping@stanford.edu.

Abstract

Biomedical studies have become larger in size and yielded large quantities of data, yet efficient data processing remains a challenge. Here we present Trellis, a cloud-based data and task management framework that completely automates the process from data ingestion to result presentation, while tracking data lineage, facilitating information query, and supporting fault-tolerance and scalability. Using a graph database to coordinate the state of the data processing workflows and a scalable microservice architecture to perform bioinformatics tasks, Trellis has enabled efficient variant calling on 100,000 human genomes collected in the VA Million Veteran Program.

Publication types

Research Support, U.S. Gov't, Non-P.H.S.