Reliability and Validity of Proposed Risk Stratification Methods for Laboratory Developed Tests

Abstract

Objectives: To determine whether different laboratory developed test (LDT) risk stratification proposals would assign differing levels of risk to selected LDTs as a measure of the validity of those proposals, and whether there would be differing interrater agreement rates as a measure of the reliability of those proposals.

Methods: A total of 4 reviewers applied 6 proposals for risk stratification of 4 LDTs. Interrater agreement was calculated as a measure of the reliability of the proposals. Also, a consensus risk categorization and concordance rate for each LDT was developed as a measure of the validity of the proposals.

Results: Interrater agreement rates (reliability) ranged from 38% to 100%, and concordance rates (validity) ranged from 20% to 100%.

Conclusions: A spectrum of reliability and validity was observed depending on the policy used and the LDT categorized. Before implementation or legislation of risk-stratification methods, large evaluations of reliability and validity should be conducted on any proposed method.

Keywords: classification; laboratory developed test; measurement; reliability; risk assessment; validity.