Chapter 14: Assessing Quality

1. Introduction

As described throughout this guide, registries are created for many purposes, including scientific, clinical, and policy, may serve more than one purpose, and may add or change purposes over time. This leads to variations in design, operations, or quality assurance that are sometimes viewed as inadequacies. It is not generally appreciated that the attributes important for some purposes may be less important for others. As a result, it is necessary to distinguish these purposes with respect to recommending particular practices.

For example, in patients for whom there is little other systematic information available, some relevant and accurate data from a registry are better than no data. Further, even registries that fall short of including all the essential elements of good registry practice described in this chapter may still provide valuable insights for some purposes. As a general rule, quality should be evaluated by elements that directly impact the ability of the registry to achieve the purpose for which it is being used. In other words, a registry should be evaluated in the context of its “fit” for a given use.

Nonetheless, there are levels of rigour that enhance validity, and make some registries more useful for generating robust evidence than others. For example, there are certain practices that enhance the validity and reliability of registries intended to be used to characterise benefits and risks and comparative effectiveness in terms of design and validity of key exposures and outcomes.

Prior to the publication of the first edition of this User’s Guide,¹ no criteria had been developed to guide evaluation of registries. Research related to the quality aspects of registries, whatever their purpose, remains relatively sparse, especially when compared with the rich information available to guide quality in clinical trials. The aim of this chapter is to provide a simple and user-friendly framework of attributes and practices that allow registries to be described and evaluated in terms of their essential elements as well as potential enhancements in the context of a given purpose. Information is presented to guide reviewers to distinguish between:

Essential registry practices that are desirable for every study.

Practices that could enhance scientific rigour and that are important for certain purposes, but may not be achievable because of practical constraints.

The items listed as “essential” elements of good practice are applicable to all patient registries. While it may not be practical or feasible to achieve all essential elements of good practice in all registries, it is useful to consider these characteristics in planning and evaluating registries.

It is also important to remind readers about some of the fundamental differences between clinical trials and registries, and how those may drive different measures of quality. For example, a clinical trial will have a rigorously maintained schedule of visits and assessment. A clinical trial patient who does not adhere to the schedule may be viewed as noncompliant. For most registries, treatments and assessments may be recommended, but are not mandated if the registry is adhering to a non-interventional design. In such cases, not every patient will have the same assessments, and assessments that are performed may be done at different time intervals, making analyses challenging. Nonetheless, some argue that the kind of data and evidence produced by these registries may be more useful for inferences needed in clinical decision making because the data reflect assessments customarily used by clinicians and the constraints — such as lack of health insurance coverage for expensive tests or treatments — experienced by both clinicians and patients in real-world situations.² Further, these registries have the added benefit of greater generalisability since few exclusion criteria are used, supporting broader inferences for many subgroups of interest.

The information described in this User’s Guide, and particularly in this chapter, is also designed for use in reporting registry study results, much like the Consolidated Standards of Reporting Trials (CONSORT) guidelines have been used for reporting of clinical trials,³ Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for observational studies in general, and the Good ReseArch for Comparative Effectiveness (GRACE) checklist for observational studies of comparative effectiveness.^4-7

2. Defining Quality

This chapter has adapted a definition of quality that was developed for randomised controlled trials.⁸ The term “quality” is used here to refer to the confidence that the design, conduct, and analysis of the registry can be shown to protect against bias (systematic error) and errors in inference — that is, erroneous conclusions drawn from a study.⁹ As used here, quality refers both to the data and to the conclusions drawn from analyses of these data. For more information about the types of biases that can affect observational studies, as well as strategies for addressing and even avoiding these biases to the extent feasible, see Chapters 3 and 13.

3. Measuring Quality

Quality must be evaluated in the context of the data elements themselves and the methods used to generate evidence. For example, high-quality data could yield lower quality evidence without appropriate curation and analyses.

Evaluations of the quality of any registry must be done with respect to the essential elements of the registry and those aspects that are important in the context of the purpose for which the registry data are being used. Both the internal and external validity of the data should be assessed, with the assessment tempered by considerations of cost, feasibility, and the context of other evidence available for the products, conditions, and target populations of interest.

The most commonly used method to assess quality of studies is a quality scale; there are numerous quality scales of varying length and complexity in existence, with strong opinions both for and against their use.^3,10Each scale emphasises distinctive dimensions of quality and therefore can yield disparate results when applied to a given study. Some scales use a summary score, derived by adding individual item scores with or without weighting. The weakness of most summary scoring approaches is that they ignore whether the items exert a bias toward the null (suggesting the erroneous interpretation that there is no effect) or tend to exaggerate the appearance of an effect when none really exists.¹¹ Furthermore, validation of the scales is difficult; studies have found wide variation in the scores for a particular study both by different reviewers and the same reviewers at different times.¹²

A quality component analysis is recommended here for assessing the quality of both the data and evidence from registries. This approach uses two domains: research quality, which pertains to the scientific process (in this instance, the design and framework of registry operations) used to generate the registry data, and evidence quality, which relates to the findings derived from the registry and processes used, including data collection, site and patient recruitment, follow-up, data curation, safety reporting, etc., in the context of a given study purpose.^13,14 According to Lohr,¹⁵ “[t]he level of confidence one might have in evidence turns on the underlying robustness of the research and the analysis done to synthesise that research.” The individual items described as essential elements of good research practice and evidence quality can be used to guide both the creation and evaluation of registries, though there are no criteria as yet as to what proportion of elements must be satisfied in order to be considered “good enough” for various purposes.

To select the quality components for analysis, key elements identified in previous research studies and quality initiatives were reviewed; among the many consulted were Guidelines for Good Pharmacoepidemiology Practice,¹⁶ the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) Guideline on Good Clinical Practice,¹⁷ the Council for International Organisations of Medical Sciences (CIOMS) International Guidelines for Ethical Review of Epidemiological Studies,¹⁸ standards developed for the conduct of registry studies for patient-centred outcomes research,¹⁹ various reports on rating scientific evidence from observational studies^10,20 and surveillance systems,^21,22 Goldberg’s review of registry evaluation methods,²³ the Meta-analysis of Observational Studies in Epidemiology (MOOSE) proposal,²⁴ the European League Against Rheumatism (EULAR) task force on biologic registers,²⁵ and Guidance for Reporting Observational Studies maintained by the Equator Network.²⁶ Special purpose quality guidance documents, including the Food and Drug Administration’s guidance document on real-world evidence,²⁷ the National Medical Device Registry Task Force report,²⁸ the Regulators Forum Registry Working Group report,²⁹ the Clinical Trials Transformation Initiative (CTTI) recommendations³⁰ for registry trials, GRACE principles for observational studies of comparative effectiveness,^6,7,31 the Registry Evaluation and Quality Standards Tool (REQueST®) to support health technology assessments,³² and other published literature^33-36 were also reviewed.

The results of the quality component analysis should be considered in terms of the registry purpose and in the context of the disease area (see Table 14-1). For example, a disease-specific registry that has been designed to look at natural history should not be deemed low quality simply because it is not large enough to detect rare or delayed treatment effects.

Table 14-1. Overview of registry purposes

Purpose	Description
Effectiveness	Determine clinical effectiveness, cost effectiveness, or comparative effectiveness of a test or treatment, including evaluating the acceptability of drugs, devices, or procedures for reimbursement.
Safety	Measure or monitor safety and harm of specific products and treatments, including delayed risks and comparative evaluation of safety and effectiveness.
Quality	Measure or improve quality of care, including conducting programmes to measure and/or improve the practice of medicine and/or public health.
Natural history	Assess natural history, including estimating the magnitude of a problem, quantifying the underlying incidence rate or prevalence of a condition or exposure; examining trends of disease over time; conducting surveillance; assessing service delivery; identifying groups who respond well or poorly to treatments, or who are at high risk; documenting the types of patients served by a health provider; and describing and estimating survival.

4. Quality Domains

Quality domains address research methods and evidence separately.

For research methods, the quality domains are design, processes and procedures which should be considered in planning, design, selection of data elements and data sources, and ethics, privacy, and governance. Table 14-2 shows the essential elements of good registry practice for research as well as the other indicators of quality that may enhance registry validity and reliability.

For evidence, the quality domains are data and research execution, and analysis and reporting, with a focus on transparency of reporting. Table 14-3 shows the essential elements of good registry practice for evidence as well as the optional further indicators of quality, including those important for selected purposes. It is important to weigh efforts to promote the accuracy and completeness of evidence in balance with the burden of reporting, the types of interventions that are available, and the risks to public health from coming to a wrong conclusion. These lists of components are most likely incomplete, but the level of detail provided should be useful for high-level quality distinctions.

Most importantly, the essential elements of good practice, as well as the optional further indicators of quality depend, to a great extent, on the resources and budget available to support registry-based research and the feasibility of collecting the data of interest.

Table 14-2. Research quality — good practice for establishing and operating registries

Domain Category	Domain	Essential Elements of Good Practice	Enhancements
Registry Design	GOALS	Develop goals, objectives and/or research questions (main and supporting, as needed). Formalise the study plan as a research protocol.	It may be helpful for external stakeholders to have input to ensure clinical relevance and feasibility.
	TARGET POPULATION	Describe the target population, including what requirements are needed to be eligible to participate in the registry and any exclusion criteria. Confirm eligibility (inclusion and exclusion criteria) upon enrolment. For studies of effectiveness and safety, use concurrent comparators, since they generally offer an advantage over historical comparison groups, especially in situations where treatments and/or diagnostics have changed over time. The comparator cohort should be as similar as possible to the exposed cohort, aside from the exposure under study.	For registries intended to study effectiveness and safety, it is often desirable to study typical patients that would use the treatments, procedure, etc. of interest. For registries where practice characteristics may influence outcome, seek to include diverse clinical practices. Where feasible, it is desirable to study diverse patients (few exclusion criteria) to facilitate analyses of subgroups.
	OBSERVATION PERIOD	Describe the follow-up time required to detect events of interest. If it is not feasible to conduct as much follow-up as desired (e.g., 20-year outcomes following hip replacement), consider whether the registry can help shape what is known about more intermediate events (e.g., 5-year outcomes following hip replacement).	Consider whether longer-term follow-up can be achieved through linkage with external data sources, and if it is feasible to collect appropriate identifiers required for accurate data linkage.
	SIZE	Determine the desired number of patients and observation time required to detect an effect should one exist, or to achieve a desired level of precision. Temper considerations about ideal study size with budgetary and feasibility constraints.	For studies of effectiveness and safety, use formal statistical calculations to estimate the number of patients or patient-years of observation needed to measure an effect with a certain level of precision or to achieve a specified statistical power to detect an effect should one exist, although the desired size may not be achievable within practical study constraints.
	DATA	Determine which variables are critical to the registry purpose and which are desirable but not critical. Focus on including those “must-have” variables that are reasonably feasible to collect and likely to be reliable, including effect-modifiers, confounders, and safety events of interest in addition to essential exposure and outcome measures. Use existing common data elements or other data standards, where appropriate, in the registry. Evaluate whether data in existing sources are of sufficient quality to achieve the registry’s purpose or if existing data can be used to supplement or minimise active data collection.	Use open standard approaches to interoperability when health information systems are used for active data collection to permit more efficient collection of data from multiple systems. Consider the collection of information to permit linkage with external databases for data supplementation (e.g., using pharmacy data to assure that prescriptions were filled) or follow-up. Use the literature to inform the choice of data elements. Consider the value of exploratory data elements for which little published literature exists.
	EXPOSURE	Determine appropriate exposure assessments to accommodate registry purposes. For studies of a specific product(s), collect sufficient information to identify the product of interest, e.g., drug or biologic brand name or generic, code, device product and its universal device identifier (UDI), etc., as appropriate and feasible.	Collect information on start and stop dates of treatments of interest and dose (if relevant) or other means to discriminate between high and low exposure.
	OUTCOMES	Choose outcomes that can be identified in typical situations and that are clinically meaningful and relevant to patients and providers. Define patient outcomes clearly, especially for conditions or outcomes that may not have uniformly established criteria (e.g., define “injection site reaction” in operational terms). Consider whether existing core sets of outcome measures or other standardised measures are available and relevant for the registry purpose, and use where feasible. Use validated scales and tests when such tools exist for the purpose needed, including measures for patient-reported outcomes. When capturing composite scores, collect and record core components, if possible. Consider where these outcomes will be collected, e.g., from medical care providers, patients, or other observers, and the likelihood that such reporters will be accurate and specific enough.	Endpoints that can be confirmed by an unbiased observer, such as death and test results, are more generalisable than endpoints that are clinical impressions without established guidance for detection and recording. Consider potential sources of error relating to accuracy and falsification of data for safety and other reporting. Any such errors should be rigorously evaluated and quantified to the extent feasible (e.g., through database and/or site reviews).
	EFFECT MODIFIERS & CONFOUNDERS	Identify important factors or characteristics that may influence response (effect modifiers or potential confounding factors), such as other important exposures (treatment), medical history, other risk factors including personal habits, and mitigating (or protective) factors. Collect these characteristics with sufficient detail for the study purpose (e.g., current smokers vs. number of packs per day and type of tobacco product for tobacco-related study purposes). Temper this list with the feasibility of data collection and burden of reporting.
	SAFETY	Consider what safety events, if any, need to be reported to satisfy regulatory requirements and develop appropriate reporting plans. Consider the timing of any such reporting obligations.	Maintain appropriate documentation, such as an audit trail, to ensure proper handling of safety information.
	ANALYSIS PLAN	Create a high-level data analysis plan to address the key objectives or research questions, e.g., how exposures and outcomes will be compared and what comparative information, if any, will be used. Determine how missing data will be handled for key variables. Describe how composite variables will be created.	Develop formal analysis plans.
Framework	ETHICS & DATA PROTECTION	Evaluate the issues of protection of human subjects — including privacy, informed consent, data security, and study ethics — and address them in accordance with local, national, and international regulations. Obtain review and approval by any required oversight committees (e.g., ethics committee, privacy committee, or institutional review board as applicable). Identify appropriate personnel and facilities, including those for secure data storage.	For any data linkage activities, determine appropriate methods for collecting and storing such protected health information. Identify the individual(s) responsible for the integrity of the data, computerised and hard copy, and make sure these individuals have the training and experience to perform the assigned tasks.
	GOVERNANCE	Develop a clear, written plan for registry governance that specifies how registry decisions will be made and describes the roles of any external advisors. Define the role of any external sponsor, including data access, use, and rights to review, participate or approve any publications.	Consider using an advisory committee(s) to guide registry objectives, data collection methods, analysis, interpretation, and dissemination. Advisory committee members who are external to the clinician, centre, or company that sponsors the research may bring added value for scientific and methodologic purposes, and may also assist with internal or external communications. If using an advisory committee, consider how decisions or recommendations will be agreed on (consensus or voting) and whether term limits and rotation would be helpful.
	TRANSPARENCY	Consider if, when, and how to allow third parties access to data, if feasible, and the process for any such data access. Assure that any data transfers are accurate, only provide the requisite data, and maintain the privacy of patients, clinicians and health systems. Plan how study results will be communicated on completion and whether the results will be made public, and if so, by whom. Consider posting information on a public registry of patient registries (e.g., at the Registry of Patient Registries).	Specify publication policies in advance of collecting data and re-evaluate as needed. It may be desirable to make key elements of the protocol, analytic methods, and results publicly accessible to promote transparency and allow other researchers to know what data might be accessible through the registry, or to consider using similar methods and study populations to confirm, refute, or extend studies derived from the registry.
	CHANGE PROCESS	Establish a process for documenting any modifications to the research plan, since the main objective(s) and analytic plans may change over time as knowledge accumulates, and the plan for data collection and follow-up may need to be adapted.	Develop plans for periodic review of analytic plans and frequency of analyses to extract the greatest value from the registry by learning through experience and remaining current with external scientific and public health advances and needs. Develop a plan for stopping or transitioning the registry, including any archiving or transferring of data and notifying participants, as appropriate.

Table 14-3. Evidence quality — indicators of good evidence quality for registries

Domain Category	Domain	Essential Elements of Good Practice	Enhancements
Methods: Data Collection, Curation, and Documentation	DATA COLLECTION	Use an efficient, reliable, and affordable means to collect data consistently of sufficient quality to meet the registry’s purpose. Prioritise simplicity and accuracy to the extent feasible. Consider using tools for automated data extraction from existing records when they are available, affordable and likely to be repeatable on a consistent basis.
	SITE AND PATIENT RECRUITMENT AND FOLLOW-UP	For primary data collection, develop plans for patient and site enrolment. Determine how to obtain follow-up data, using similar methods for all subjects in each group, to the extent feasible. Plan to expend reasonable efforts to ensure that appropriate patients are enrolled and followed systematically. Methods used for follow-up, including efforts to minimise loss to follow-up, should be documented. Use similar methods for following all study subjects to the extent feasible.	Depending on study purpose, it may be desirable to include researchers with a range of experience and expertise. A feasibility study or pilot test (e.g., when studying hard-to-reach populations, when sensitive data are sought, and when critical registry methods are new or have not otherwise been tested) can be useful to assure that the study plan is feasible and sufficiently appealing to contributors and participants. Enrolment logs are useful to document sequential enrolment. Loss to follow-up should be evaluated periodically to see if there is differential loss to follow-up, and, if it is remediable.
	DATA COLLECTION GUIDANCE	Methods for data collection should be documented. Provide clear, operational definitions of outcomes and other data elements. Documentation should provide explicit definitions of data elements and their coding. Develop standard instructions for use in training data collectors. For safety studies, create a process for identifying and reporting serious events that is consistent with regulatory requirements. For studies with direct patient contact, plan training for study personnel about how to identify or recognise serious events, including: (1) asking about complaints or adverse events in a manner that is clear and specific (e.g., solicited vs. unsolicited), and (2) knowing if, how and when information should be reported to manufacturers and health authorities. For studies using existing data sources, use uniform and systematic methods for collecting and curating the data to assure that the appropriate data have been extracted and linked (where applicable).	Whenever possible, use standardised data dictionaries, such as the International Classification of Diseases, and use coding that is consistent with nationally or internationally approved coding systems to promote comparability of information among studies. Methods used for data transformations should be recorded. Uniform written guidance for any active medical chart review and abstraction will enhance validity and reliability. For studies linking to or integrating existing data sources, document the process for record linkage and whether probabilistic or deterministic matching was used. Consider any additional requirements that may influence successful linkage, such as selection of data elements used to assure accurate matching.
	QUALITY ASSURANCE	Develop a data handling and analysis plan that describes any quality assurance and data curation activities that will be implemented. Any quality assurance procedures used must be fit for purpose and should be focused on variables that are essential for analysis, such as endpoints of primary interest. Data checks should use range and consistency checks and may also include a review of consistency and comparability of data across sites, and with external data sources. Methods should be described for data curation, e.g., quality control procedures to enhance internal validity, review of consistency and comparability of data across sites, and any comparisons that will be made with external data sources.	Quality assurance (QA) may include review or monitoring of a sample of data and/or data review by an adjudication committee for complex conditions or endpoints for which established procedures and/or coding are not used. For primary data collection, a sample of data collected should be compared with patient records (e.g., 5–20% of patients’ records) to assure validity and reproducibility of abstraction and coding. For some studies and some outcomes, validation of endpoints may be desirable depending on the study purpose (e.g., may be necessary for some regulatory uses). If the registry chooses to implement a system of periodic monitoring for quality assurance, a risk-based strategy should be used to focus on detecting and quantifying the most likely causes of error and the types of error most likely to affect the registry purpose, with QA activities adapted based on observed performance (e.g., increase QA for sites that appear to be having difficulty in study conduct or data entry). Establish processes and standards for creating analytic data files and maintaining such files to support publications and presentations, since registry analyses may be performed on live data (data that may change as the registry continues to collect and verify information through various quality control procedures) or on data that have been locked and have undergone formal review and editing.
Reporting	OVERALL REPORTING	Registry reports or publications should describe the methods, including target population and selection of sites and study subjects, compliance with applicable regulatory rules and regulations, data collection and curation/quality control methods, statistical methods for data analysis, and any circumstances that may have affected the quality or integrity of the data. The information should be reported with enough detail to allow replication of the methods in another study. Follow-up time should be described to enable assessment of the impact of the observation period on the conclusions drawn.	Completeness of information on eligible patients should be evaluated and described for key variables of interest for the main exposures and/or outcomes of primary interest.
	ANALYTICS	Results should be reported for all the main objectives, including estimates of effect for each (where relevant) and confidence intervals where feasible. The data elements used in any models should be described. For safety studies, the risks and/or benefits of products, devices, or processes under study should be quantitatively evaluated beyond simply evaluating statistical significance (e.g., rates, proportions, and/or relative risks, as well as confidence intervals, were reported). The role and impact of missing data and potential confounding factors should be considered. The findings should be compared and contrasted with other relevant research.	Information about data analyses, including transformation of variables and/or construction of composite endpoints and how missing data were handled, should be reported with sufficient detail to permit replication. Appropriate analytic approaches should be used to address confounding. Sensitivity analyses should be used to examine and quantify the potential impact on the association between the exposure(s) of interest and the outcome(s) by, for example, quantifying how the results would change if all the missing data were the same, e.g., respondents lied about medical adherence or certain risk factors likely to have a strong effect on the outcome, or all missing data on smokers was from people who smoke. Selection bias should be evaluated by describing the representativeness of the actual population in terms of how it was selected, how well the characteristics of the actual population match those of the target population, and to whom the results apply.
	COMPARISONS	For comparative studies, comparators reflect medical practice for the appropriate reference time period. When other reasonably accurate and relevant contemporaneous comparative data are not available, historical data should be used with appropriate justification. Contemporaneous comparators in similar health systems or sites are preferable when feasible.	External validity should be described by showing how registry participants compare to known characteristics of the target population in terms of key characteristics and other factors that may be used to characterise the patients, providers and healthcare settings that were studied.

References for Chapter 14

1. Gliklich RE, Dreyer NA, editors. Registries for Evaluating Patients Outcomes: A User’s Guide. Rockville, MD: Agency for Healthcare Research and Quality; Apr, 2007.	2. Dreyer NA, Garner S. Registries for robust evidence. JAMA. 2009;302(7):790-1. PMID: 19690313. DOI: 10.1001/jama.2009.1092.
3. Moher D, Schulz KF, Altman D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285(15):1987-91. PMID: 11308435.	4. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495-9. PMID: 25046131. DOI: 10.1016/j.ijsu.2014.07.013.
5. Vandenbroucke JP, von Elm E, Altman DG, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4(10):e297. PMID: 17941715. DOI: 10.1371/journal.pmed.0040297.	6. Dreyer NA, Bryant A, Velentgas P. The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. J Manag Care Spec Pharm. 2016;22(10):1107-13. PMID: 27668559. DOI: 10.18553/jmcp.2016.22.10.1107.
7. Dreyer NA, Schneeweiss S, McNeil BJ, et al. GRACE principles: recognizing high-quality observational studies of comparative effectiveness. Am J Manag Care. 2010;16(6):467-71. PMID: 20560690.	8. Moher D, Jadad AR, Nichol G, et al. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1):62-73. PMID: 7743790.
9. Rothman K, editor. Causal inference. Chestnut Hill, MA: Epidemiology Resources, Inc.; 1988.	10. West S, King V, Carey TS, et al. Systems to rate the strength of scientific evidence. Evid Rep Technol Assess (Summ). 2002(47):1-11. PMID: 11979732.
11. Greenland S. Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol. 1994;140(3):290-6. PMID: 8030632. DOI: 10.1093/oxfordjournals.aje.a117248.	12. Berkman ND, Lohr KN, Ansari M, et al. Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008.
13. Mosteller F, Boruch R, editors. Evidence matters: randomized trials in education research. Washington, DC: The Brookings Institute; 2002.	14. Shavelson RJ, Towne L, editors. Scientific research in education. Washington, DC: National Research Council, National Academy Press; 2002.
15. Lohr KN. Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care. 2004;16(1):9-18. PMID: 15020556. DOI: 10.1093/intqhc/mzh005.	16. ISPE. Guidelines for good pharmacoepidemiology practices (GPP). Pharmacoepidemiol Drug Saf. 2008;17(2):200-8. PMID: 17868186. DOI: 10.1002/pds.1471.
17. European Medicines Agency. ICH Topic E6(R2) Guideline for Good Clinical Practice. https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-6-r1-guideline-good-clinical-practice-step-5_en.pdf. Accessed June 10, 2019.	18. CIOMS. 1991 International Guidelines for Ethical Review of Epidemiological Studies. http://www.cioms.ch/publications/guidelines/1991_texts_of_guidelines.htm. Accessed June 10, 2019.
19. Gliklich R, Dreyer NA, Leavy MB, et al. Standards in the conduct of registry studies for patient-centered outcomes research. Prepared for the Patient-Centered Outcomes Research Institute, Washington, DC, 2012. http://www.pcori.org/assets/Standards-in-the-Conduct-of-Registry-Studies-for-Patient-Centered-Outcomes-Research.pdf. Accessed June 10, 2019.	20. Shea BJ, Hamel C, Wells GA, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013-20. PMID: 19230606. DOI: 10.1016/j.jclinepi.2008.10.009.
21. Centers for Disease Control. Guidelines for evaluating surveillance systems. MMWR Suppl. 1988;37(5):1-18. PMID: 3131659.	22. German RR, Lee LM, Horan JM, et al. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recomm Rep. 2001;50(RR-13):1-35. PMID: 18634202.
23. Goldberg J, Gelfand HM, Levy PS. Registry evaluation methods: a review and case study. Epidemiol Rev. 1980;2:210-20. PMID: 7000537. DOI: 10.1093/oxfordjournals.epirev.a036224.	24. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. MOOSE group. JAMA. 2000;283(15):2008-12. PMID: 10789670.
25. Dixon WG, Carmona L, Finckh A, et al. EULAR points to consider when establishing, analysing and reporting safety data of biologics registers in rheumatology. Ann Rheum Dis. 2010;69(9):1596-602. PMID: 20525843. DOI: 10.1136/ard.2009.125526.	26. Equator Network. http://www.equator-network.org/. Accessed June 10, 2019.
27. U.S. Food and Drug Administration. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices. Guidance for Industry and Food and Drug Administration Staff. https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM513027.pdf. Accessed June 3, 2019.	28. Recommendations for a National Medical Device Evaluation System. A Report from the Medical Device Registry Task Force & the Medical Devices Epidemiology Network. August 20, 2015. http://mdepinet.org/wp-content/uploads/2015-08-Recommendations-for-a-National-Medical-Device-Evaluation-System.pdf. Accessed June 10, 2019.
29. International Medical Device Regulators Forum. Patient Registry: Essential Principles. October 2, 2015. http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-essential-principles-151124.pdf. Accessed June 10, 2019.	30. Clinical Trials Transformation Initiative. CTTI Recommendations: Registry Trials. May 2017. https://www.ctti-clinicaltrials.org/files/recommendations/registrytrials-recs.pdf. Accessed June 10, 2019.
31. Dreyer NA, Velentgas P, Westrich K, et al. The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution. J Manag Care Spec Pharm. 2014;20(3):301-8. PMID: 24564810. DOI: 10.18553/jmcp.2014.20.3.301.	32. Registry Evaluation and Quality Standards Tool (REQueST®). European Network for Health Technology Assessment (EUnetHTA). https://www.eunethta.eu/request-tool-and-its-vision-paper/. Accessed February 13, 2020.
33. National Academies of Sciences, Engineering, and Medicine. 2017. Real-world evidence generation and evaluation of therapeutics: Proceedings of a workshop. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/24685.	34. Berger M, Daniel G, Frank K., et al. A Framework for Regulatory Use of Real-World Evidence. The Margolis Center for Health Policy at Duke University. https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf. Accessed June 10, 2019.
35. Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017;26(9):1033-9. PMID: 28913966. DOI: 10.1002/pds.4297.	36. Miksad RA, Abernethy AP. Harnessing the Power of Real-World Evidence (RWE): A Checklist to Ensure Regulatory-Grade Data Quality. Clin Pharmacol Ther. 2018;103(2):202-5. PMID: 29214638. DOI: 10.1002/cpt.946.