Lot quality assurance sampling saves time, money while providing valuable information on HIV services

 David Boone The following is a guest post by David Boone, Ph.D., of MEASURE Evaluation Most HIV monitoring and evaluation folks are familiar with the challenges of keeping track of the number of patients currently on treatment, and the data quality problems that result from poor record keeping. Patients need to be followed perpetually, and they come and go from treatment rosters all the time. It requires an enormous effort to maintain accurate records and to do data quality assessments because the status of each patient needs verified to update the number of patients currently on antiretroviral therapy  in each facility. At MEASURE Evaluation, funded by the United States Agency for International Development, we tested a method in Burundi in May to June 2019 to sample patient records, assess them for particular criteria, and then make judgements on data quality for all records in the facility for that service, based on the results of the sample. Using lot quality assurance sampling we are able to take a sample of around 50–70 records, (depending on the size of the cohort on treatment) and determine with quantifiable levels of confidence and error whether the facility “passes” or “fails” to meet a predetermined standard. Facilities that fail to meet the standard are then targeted for more exhaustive reviews to clean up records and fill gaps. Facilities that pass are deemed “good to go” until the next round of supervision. Eventually, all sites with data quality issues will be identified and corrected. We believe this method has great potential as a cost and resource saver because sites that pass will be spared the expense and burden of regular exhaustive reviews of data quality. The sample sizes used in lot quality assurance sampling are too small to allow for point estimates (e.g., 95% of clients have had viral load test done and viral load is suppressed). However, the patient records for a given service (the ‘lot’ in lot quality assurance sampling) can be classified as meeting or not meeting a predetermined target, and performance of different facilities can be compared. Setting this target is a crucial step in lot quality assurance sampling, where the main idea is to identify the worst of the worst in order to prioritize resources for improving data management. In Burundi we conducted a traditional data quality assessment of “current patients on ART” and “patients with viral load test done and viral load suppressed.” Because we were conducting exhaustive reviews of patient medical records and other data sources, we were able to use the lot quality assurance sampling method and compare its findings with those of the exhaustive review — a means of validating the accuracy of the sampling method.  For “current on ART,” we conducted this test in around 40 facilities; and for “viral load” we were able to test 15 facilities (not all facilities in Burundi are conducting viral load tests or recording results in source documents). We set a quality threshold for three criteria: completeness of data, concordance between data sources, and outcomes for “current on treatment” and “viral load.” With certain inputs (quality threshold, tolerable error rates, and patient volume in the sites) we were able to determine a sample size requirement for each facility using an online sample size calculator. On the comparison between lot quality assurance sampling results and the exhaustive review for both “current on ART” and “viral load,” we found around 75% agreement, which, while not spectacular, is promising enough to warrant further trials. In the Burundi experience, we learned that the assessment of outcomes was hindered by lack of completeness in source documents (i.e., medical records, registers, and the electronic patient tracking system). Also, inconsistent recording practices by data management staff at facilities made it difficult to accurately record outcomes for all patients in the sample. At MEASURE Evaluation, we believe the so-called ‘lot quality assurance sampling triage system’ is a valuable complement to traditional data quality assurance methods, particularly for high-priority indicators for which accuracy is essential (e.g., current on ART). When used as part of a routine system of health facility supervision, this approach can save time, effort, and resources while yielding statistically sound results with quantifiable confidence and error. It can improve data in source documents allowing for improved patient management and, because data quality issues will be identified and resolved at the source, aggregate data reported to national programs will be more accurate. An Excel-based tool and companion guidelines document for the lot quality assurance sampling Triage System can be found at https://www.measureevaluation.org/resources/publications/ms-19-176 For more information about MEASURE Evaluation, visit www.measureevaluation.org David Boone is an epidemiologist with MEASURE Evaluation, John Snow, Inc.