Background: The study was carried out in order to assess the inter-observer reliability and validity in respect of clinical appraisal given by the Appropriateness Evaluation Protocol (AEP), in the context of the Spanish Public Hospital System.
Material and method: In order to assess the reliability a total of 614 hospital stays chosen at random from 56 hospital admissions were independently analysed by three reviewers (two doctors and one nurse). In order to assess the validity, the findings obtained by the nurse were compared with the majority opinion given by the 7 hospital specialists in respect of each of hospital stays under evaluation. As part of the analytical procedure, indices for observed agreement, and specific agreement were calculated, as well as the Kappa statistic, all forming of various random samples of 614 hospital stays. In order to assess the predictive validity of the AEP, its sensitivity, specificity and predictive values were all measured against the majority clinical judgement.
Results: The study exhibited a high degree of inter-observer reliability (specific agreement > 64%, kappa > 0.75) and a reasonable validity in comparison with the consensus of opinions formed by a least 4 or 5 of its 7 clinical reviewers (specific agreement > 61%, kappa > 0.64), these values decreasing notably when the consensus of 6 or 7 of the reviewers was required. The AEP revealed a high degree of sensitivity and a low degree of specificity in comparison with the majority clinical assessment, thus minimising the occurrence of false results when the stay was regarded as appropriate, and producing false negatives (appropriate hospital stays regarded as inappropriate) varying in degree from moderate to very high.
Conclusions: The results showing high reliability and moderate validity regarding clinical assessment shows the AEP to be a useful instrument in the sifting-out of inappropriate use of hospitalisation, although they do not allow a definitive judgement to be made concerning the efficiency of hospital services nor judgements regarding individual cases.