Pre Content

Registries for Evaluating Patient Outcomes:

A User's Guide

Prepared for:

Agency for Healthcare Research and Quality

U.S. Department of Health and Human Services

5600 Fishers Lane Rockville, MD 20857

www.ahrq.gov

Contract No.290-2014-00004-C

Senior Editors:

Richard E. Gliklich, M.D. Chief Executive Officer and Chairman OM1

Michelle B. Leavy, M.P.H. Head, Healthcare Research & Policy OM1

Nancy A. Dreyer, Ph.D., M.P.H., FISPE Chief Scientific Officer IQVIA Real-World & Analytic Solutions

AHRQ Publication No. 19(20)-EHC020

September 2020

The Effective Health Care Program of the Agency for Healthcare Research and Quality (AHRQ) conducts and supports research focused on the outcomes, effectiveness, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and healthcare services. More information on the Effective Health Care Program can be found at www.effectivehealthcare.ahrq.gov. This report was produced under contract to the Agency for Healthcare Research and Quality, Rockville, MD by L&M Policy Research, LLC, and partners OM1 and IQVIA under Contract No. 290-2014-00004-C. The AHRQ Task Order Officer for this project was Elise Berliner, Ph.D.

The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services. All statements in this report, including its findings and conclusions, are solely those of the authors. The authors participated as individuals and not necessarily as representatives of their organizations. No statement in this document should be construed as an official position of any organization represented here.

©2020 United States Government, as represented by the Secretary of the Department of Health and Human Services, by assignment. All rights reserved. The Agency for Healthcare Research and Quality (AHRQ) permits members of the public to reproduce, redistribute, publicly display, and incorporate this work into other materials provided that it must be reproduced without any changes to the work or portions thereof, except as permitted as fair use under the U.S. Copyright Act. This work contains certain tables and figures noted herein that are subject to copyright by third parties. These tables and figures may not be reproduced, redistributed, or incorporated into other materials independent of this work without permission of the third-party copyright owner(s). This work may not be reproduced, reprinted, or redistributed for a fee, nor may the work be sold for profit or incorporated into a profit-making venture without the express written consent of AHRQ. This work is subject to the restrictions of Section 1140 of the Social Security Act, 42 U.S.C. § 1320b-10. When parts of this work are used or quoted, the following citation should be used:

Citation

Gliklich RE, Leavy MB, Dreyer NA (sr eds). Registries for Evaluating Patient Outcomes: A User's Guide. 4th ed. (Prepared by L&M Policy Research, LLC, under Contract No. 290-2014-00004-C with partners OM1 and IQVIA) AHRQ Publication No. 19(20)- ^{EHC020. Rockville, MD: Agency for Healthcare Research and Quality;} September 2020. Posted final reports are located on the Effective Health Care Program search page. DOI: https://doi.org/10.23970/AHRQEPCREGISTRIES4.

Acknowledgments

Numerous individuals contributed to the coordination and formatting of this document. In addition to the authors and reviewers, we wish to acknowledge the following individuals for their significant time and dedication: Devyn Andrews of OM1, Monica Sarmiento of L&M Policy Research, and Elise Berliner and Chris Heidenrich of the Agency for Healthcare Research and Quality.

Preface

This project was performed under a contract from the Agency for Healthcare Research and Quality (AHRQ) with the purpose of updatingRegistries for Evaluating Patient Outcomes: A User's Guide. First published in 2007, the User's Guide, with translations available in Chinese and Korean, serves as a reference for planning, developing, maintaining, and evaluating registries designed to collect data about patient outcomes. The second (2010) and third (2014) editions incorporated updates to existing topics and included new chapters on methodological and technological advances in registry science. The first addendum to the Third Edition,21^st Century Patient Registries, was published in March 2018 and addressed approaches to engaging patient partners in registry design and operations. The second addendum to the Third Edition, Tools and Technologies for Registry Interoperability, was published in December 2019 and addressed concepts related to incorporating other data sources into patient registries.

The purpose of this revised fourth edition is to incorporate information on new methodological and technological advances into the existing chapters and to consolidate and organize the content into a format that emphasizes the key principles of registry design, operations, and analysis. Two new chapters were added (Registry Governance and Selecting and Defining Outcome Measures for Registries), and existing chapters were updated as part of this effort. Like the prior editions of the User's Guide, this fourth edition was created with support from a large group of stakeholders representing academia, industry, government, patient organizations, and technology organizations. At the outset, we solicited feedback on chapter topics and outlines from AHRQ, academics, and other experts in the field. We then reached out to topic experts inviting participation in writing or reviewing the final topics selected. Once the authorship groups were established, many meetings were held to update the chapters prior to sending them for constructive feedback and editorial review to the assigned reviewer group for each paper. The collaborative efforts of contributors, reviewers, and editors resulted in a draft document that was posted for public comment on the Effective Health Care website in April 2019. This document incorporates much of the feedback received. Like previous editions, the contributors and reviewers participated as individual experts and not necessarily as representatives of their organizations. We are grateful to all those who contributed in writing, reviewing and editing this document.

To begin the discussion of registries, we would like to clarify some distinctions between registries and clinical trials. Although this subject is discussed further in Chapter 1, we offer here the following distinctions from a high-level perspective. A clinical trial is an experiment in which an active intervention intended to change a human subject's outcome is implemented, generally through a randomization procedure that takes decision making away from the practitioner. The research protocol describes inclusion and exclusion criteria that are used to select the patients who will participate as human subjects, focusing the experiment on a homogeneous group. Human subjects and clinical researchers agree to adhere to a strict schedule of visits and to conduct protocol-specific tests and measurements.

In contrast, registries use an observational study design that does not specify treatments or require any therapies intended to change patient outcomes (except insofar as specific treatments or therapies may be inclusion criteria). Inclusion and exclusion criteria are kept to a minimum in an effort to study a broad range of patients in order to make the results more generalizable.

Patients are typically observed as they present for care, and the data collected generally reflect whatever tests and measurements a provider customarily uses.

Patient registries represent a useful tool for a number of purposes. Their ideal use and their role in evidence development, design, operations, and evaluation resemble but differ from clinical trials in a number of substantive ways, and therefore they should not be evaluated with the same constructs. This user's guide presents what the contributors and reviewers consider good registry practices. Many registries today may not meet even the basic practices described. On the whole, registry science is in an active state of development. This fourth edition of the user's guide is an important step in developing the field.

This book is divided into four sections: Creating Registries; Legal and Ethical Considerations for Registries; Operating Registries; and Evaluating Registries. The first three sections provide basic information on key areas of registry development and operations, highlighting the spectrum of practices in each of these areas and their potential strengths and weaknesses.

Section I, "Creating Registries," contains six chapters. "Patient Registries" defines and characterizes types of registries, their purposes, and uses, and describes their place within the scope of this document. "Planning a Registry" focuses on the recommended steps in planning a registry, from determining if a registry is the right option to describing goals and objectives. "Registry Design" examines the specifics of designing a registry once the goals and objectives are known. "Selecting and Defining Outcome Measures for Registries" describes considerations related to identifying the most relevant outcomes to measure and selecting appropriate definitions for these outcomes. "Data Elements for Registries" provides a scientific and practical approach to selecting data elements. "Data Sources for Registries" describes how existing data sources (administrative, pharmacy, other registries, etc.) may be used to enhance the value of patient registries.

Section II, "Legal and Ethical Considerations for Registries," contains three chapters. "Principles of Registry Ethics, Data Ownership, and Privacy" reviews several key legal and ethical issues that should be considered in creating or operating a registry. "Informed Consent for Registries" discusses how the requirements of informed consent for patient registries differ from those for clinical trials and offers suggestions for creating informed consent documents that address the unique aspects of registries. "Registry Governance" describes considerations that should inform plans for registry governance and provides examples of potential governance structures.

Section III, "Operating Registries," provides a practical guide to the day-to-day operational issues and decisions for producing and interpreting high-quality registries. "Recruiting and Retaining Participants in the Registry" describes strategies for recruiting and retaining providers and patients. "Obtaining Data and Quality Assurance" reviews key areas related to obtaining, cleaning, and storing data and quality assurance for registries. "Adverse Event Detection, Processing, and Reporting" examines relevant practical and regulatory issues. "Analysis, Interpretation, and Reporting of Registry Data To Evaluate Outcomes" addresses key considerations in analyzing and interpreting registry data.

Section VI is "Evaluating Registries." This final chapter on "Assessing Quality" summarizes key points from the earlier chapters in a manner that can be used to review the structure, data, or

interpretations of patient registries. It describes good registry practice in terms of "essential elements" and "further indicators of quality." This information might be used by a person developing a registry, or by a reviewer or user of registry data or interpretations derived from registries.

Richard E. Gliklich Michelle B. Leavy Nancy A. Dreyer Editors

Executive Summary

Defining Patient Registries

This User's Guide is intended to support the design, implementation, analysis, interpretation, and quality evaluation of registries created to increase understanding of patient outcomes. For the purposes of this guide, a patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves one or more stated scientific, clinical, or policy purposes. A registry database is a file (or files) derived from the registry. Although registries can serve many purposes, this guide focuses on registries created for one or more of the following purposes: to describe the natural history of disease, to determine clinical effectiveness or cost-effectiveness of healthcare products and services, to measure or monitor safety and harm, and/or to measure quality of care.

Registries are classified according to how their populations are defined. For example, product registries include patients who have been exposed to biopharmaceutical products or medical devices. Health services registries consist of patients who have had a common procedure, clinical encounter, or hospitalization. Disease or condition registries are defined by patients having the same diagnosis, such as cystic fibrosis or heart failure.

Planning a Registry

There are several key steps in planning a patient registry, including articulating its purpose, determining whether it is an appropriate means of addressing the research question, identifying stakeholders, defining the scope and target population, assessing feasibility, and securing funding. The registry team and advisors should be selected based on their expertise and experience. The plan for registry governance and oversight should clearly address such issues as overall direction and operations, scientific content, ethics, safety, data access, publications, and change management. It is also helpful to plan for the entire lifespan of a registry, including how and when the registry will end and any plans for transition at that time. Special consideration should be given to the unique challenges of planning specific types of registries, such as rare disease registries or quality improvement registries.

Registry Design

A patient registry should be designed with respect to its major purpose, with the understanding that different levels of rigor may be required for registries designed to address focused analytical questions to support decision making, in contrast to registries intended primarily for descriptive purposes. The key points to consider in designing a registry include formulating a research question; choosing a study design; translating questions of clinical interest into measurable exposures and outcomes; choosing patients for study, including deciding whether a comparison group is needed; determining where data can be found; and deciding how many patients need to be studied and for how long. Once these key design issues have been settled, the registry design should be reviewed to evaluate potential sources of bias (systematic error); these should be addressed to the extent that is practical and achievable. The information value of a registry is enhanced by its ability to provide an assessment of the potential for bias and to quantify how this bias could affect the study results.

The specific research questions of interest will guide the registry's design, including the choice of exposures and outcomes to be studied and the definition of the target population (the population to which the findings are meant to apply). The registry population should be designed to approximate the characteristics of the target population as much as possible. The number of study subjects to be recruited and the length of observation (followup) should be planned in accordance with the overall purpose of the registry. The desired study size (in terms of subjects or person-years of observation) is determined by specifying the magnitude of an expected, clinically meaningful effect or the desired precision of effect estimates. Study size determinants are also affected by practicality, cost, and whether the registry is intended to support regulatory decision making. Depending on the purpose of the registry, internal, external, or historical comparison groups strengthen the understanding of whether the observed effects are indeed real and in fact different from what would have occurred under other circumstances. Registry study designs often restrict eligibility for entry to individuals with certain characteristics (e.g., age) to ensure that the registry will have subgroups with sufficient numbers of patients for analysis. Or the registry may use some form of sampling—random selection, systematic sampling, or a haphazard, nonrandom approach—to achieve this end.

Special consideration should be given to the unique challenges of designing registries for specific purposes, such as product safety surveillance, rare diseases, medical devices, and quality improvement.

Selecting and Defining Outcome Measures for Registries

The selection and definition of patient outcomes of interest is a critical step in designing a patient registry. The outcomes of interest, together with the exposures(s) of interest, drive many of the decisions regarding the study duration, the necessary data elements, and the source(s) of the data. Outcomes should be selected primarily based on the research questions of interest, with consideration given to the feasibility of capturing the desired outcomes within the study scope and budget. It is also important to consider the perspectives of multiple stakeholders when determining which outcomes are most relevant. Tools such as the Outcomes Measures Framework can be helpful to guide the selection and definition of outcome measures for use within registries. In addition, the use of standardized outcome measures or other data standards, when available, is essential so that registries can maximally contribute to evolving medical knowledge. Standard terminologies—and to a greater degree, higher level groupings into core datasets for specific conditions—not only improve efficiency in establishing registries but also promote more effective sharing, combining, or linking of datasets from different sources. Furthermore, the use of well-defined standards for data elements and data structure ensures that the meaning of information captured in different systems is the same. This is critical to maximize the value of registries as tools in learning health systems and a national research infrastructure.

Data Elements

The selection of data elements requires balancing such factors as their importance for the integrity of the registry and for the analysis of primary outcomes, their reliability, their contribution to the overall burden for respondents, and the incremental costs associated with their collection. Selection begins with identifying relevant domains. Specific data elements are then selected with consideration for established clinical data standards, common data definitions, and whether patient identifiers will be used. It is important to determine which elements are absolutely necessary and which are desirable but not essential. In choosing measurement scales for the assessment of patient-reported outcomes, it is preferable to use scales that have been appropriately validated, when such tools exist. Once data elements have been selected, a data map should be created, and the data collection tools should be pilot tested. Testing allows assessment of respondent burden, the accuracy and completeness of questions, and potential areas of missing data. Inter-rater agreement for data collection instruments can also be assessed, especially in registries that rely on chart abstraction. Overall, the choice of data elements should be guided by parsimony, validity, and a focus on achieving the registry's purpose.

Data Sources

A single registry may integrate data from various sources. The form, structure, availability, and timeliness of the required data are important considerations. Data sources can be classified as primary or secondary. Primary data are collected by the registry for its direct purposes. Secondary data have been collected by a secondary source for purposes other than the registry, and may not be uniformly structured or validated with the same rigor as the registry's primary data. Sufficient identifiers are necessary to guarantee an accurate match between data from secondary sources and registry patients. Furthermore, it is advisable to obtain a solid understanding of the original purpose of the secondary data, because the way those data were collected and verified or validated will help shape or limit their use in a registry. Common secondary sources of data linked to registries include medical records systems, institutional or organizational databases, administrative health insurance claims data, death and birth records, census databases, and related existing registry databases.

Ethics, Data Ownership, and Privacy

Critical ethical and legal considerations should guide the development and use of patient registries. The Common Rule is the uniform set of regulations on the ethical conduct of human subjects research, issued by the Federal agencies that fund such research. Institutions that conduct research agree to comply with the Common Rule for federally funded research, and may opt to apply that rule to all human subjects activities conducted within their facilities or by their employees and agents, regardless of the source of funding. The Privacy Rule, promulgated under the Health Insurance Portability and Accountability Act of 1996 (HIPAA), establishes Federal protections for the privacy of individually identifiable health information created and maintained by health plans, healthcare clearinghouses, and most healthcare providers (collectively, "covered entities"). The purpose of a registry, the type of entity that creates or maintains the registry, the types of entities that contribute data to the registry, and the extent to which registry data are individually identifiable affect how the regulatory requirements apply. Other important concerns include transparency of activities, oversight, and data ownership. This chapter of the User's Guide focuses solely on U.S. law. Health information is also legally protected in European and some other countries by distinctly different rules.

The requirement of informed consent often raises different issues for patient registries versus clinical trials. For example, registries may be used for public health or quality improvement activities, which may not constitute "human subjects research." Also, registries may integrate data from multiple electronic sources (e.g., claims data, electronic health records) and may be linked to biobanks. Institutional review boards may approve waivers or alterations of informed consent (e.g., electronic consent, oral consent) for some registries, depending on the purpose and risk to participants. Established registries that undergo a change in scope (e.g., changes in data sharing policies, changes to the protocol, extension of the followup period) may need to ask patients to "re-consent." When planning informed consent procedures, registry developers should consider several factors, including documentation and format, consent revisions and re-consent, the applicability of regulatory requirements, withdrawal of participants from the study, and the physical and electronic security of patient data and biological specimens. In addition, registry developers may need to consider the individual authorization requirements of the HIPAA Privacy Rule, where applicable.

Registry Governance

Registries function in a dynamic environment and are often shaped by the complex relationships among individual health, public health policy, economics, geography, and culture. Complexity within registries stems from the topics being studied, stakeholders with different agendas, and the legal and political climates for such research, among other factors. Governance is an important tool to help registries manage complexities such as these across the registry lifecycle, from the initial planning phase through the dissemination of results. Registry governance refers to a formalized structure or plan for managing the registry and guiding decision making related to registry funding, operations, and dissemination of information. Registry governance can take many forms depending on the scope of the registry, the number of stakeholders, and the purpose of the registry, but some principles for successful governance apply across all governance models. In particular, all aspects of governance should be codified in a written format that can be reviewed, shared, and refined over time, and transparency regarding any perceived or actual conflicts of interest is important for effective governance. Expectations of each research partner should be clearly delineated, pragmatic, and transparent. Lastly, policies and procedures should be developed to support stakeholder engagement and transparency.

Patient and Provider Recruitment and Management

Recruitment and retention of patients as registry participants, and of providers as registry sites, are essential to the success of a registry. Recruitment typically occurs at several levels, including facilities (hospitals, physicians' practices, and pharmacies), providers, and patients. The motivating factors for participation at each level and the factors necessary to achieve retention differ according to the registry. Factors that motivate participation include the perceived relevance, importance, or scientific credibility of the registry, as well as a favorable balance of any incentives for participation versus the risks and burdens thereof. Because patient and provider recruitment and retention can affect how well a registry represents the target population, well-planned strategies for enrollment and retention are critical. Goals for recruitment, retention, and followup should be explicitly laid out in the registry planning phase, and deviations during the conduct of the registry should be continuously evaluated for their risk of introducing bias.

Obtaining Data and Quality Assurance

The integrated system for obtaining, cleaning, storing, monitoring, reviewing, and reporting on registry data determines the utility of those data for meeting the registry's goals. A broad range of procedures and systems are available for obtaining or collecting data. Some are more suitable than others for particular purposes. Critical factors in the ultimate quality of the data include how data elements are structured and defined, how personnel are trained, and how data problems (e.g., missing, out of range, or logically inconsistent values) are handled. Registries may also be required to conform to guidelines or to the standards of specific end users of the data (e.g., 21 Code of Federal Regulations, Part 11). Quality assurance aims to affirm that the data were, in fact, collected in accordance with established procedures and that they meet the requisite standards of quality to accomplish the registry's intended purposes and the intended use of the data. Requirements for quality assurance should be defined during the registry's inception and creation. Because certain requirements may have significant cost implications, a risk-based approach to developing a quality assurance plan is recommended. It should be based on identifying the most important or likely sources of error or potential lapses in procedures that may affect the quality of the registry in the context of its intended purpose.

Adverse Event Detection, Processing, and Reporting

The U.S. Food and Drug Administration defines an adverse event (AE) as any untoward medical occurrence in a patient administered a pharmaceutical product, whether or not related to or considered to have a causal relationship with the treatment. AEs are categorized according to the seriousness and, for drugs, the expectedness of the event. Although AE reporting for all marketed products is dependent on the principle of "becoming aware," collection of AE data falls into two categories: those events that are intentionally solicited (meaning data that are part of the uniform collection of information in the registry) and those that are unsolicited (meaning that the AE is volunteered or noted in an unsolicited manner). The determination of whether the registry should use a case report form to collect AEs should be based on the scientific importance of the information for evaluating the specified outcomes of interest. Regardless of whether or not AEs constitute a primary objective of the registry, it is important for any registry that has direct patient interaction to develop a plan for detecting, processing, and reporting AEs. If the registry receives sponsorship, in whole or in part, from a regulated industry (drugs or devices), the sponsor has mandated reporting requirements including stringent timelines, and the registry should establish the process for detecting and reporting AEs and should provide training to registry personnel on how to identify AEs and to whom they should be reported. Sponsors of registries designed specifically to meet requirements for surveillance of drug or device safety are encouraged to hold discussions with health authorities about the most appropriate process for reporting serious AEs.

Analysis, Interpretation, and Reporting of Registry Data

Analysis and interpretation of registry data begin with answering a series of core questions: Who was studied, and how were they chosen for study? How were the data collected, edited, and verified, and how were missing data handled? How were the analyses performed? Four populations are of interest in describing who was studied: the target population, the accessible population, the intended population, and the population actually studied (the "actual population"). The representativeness of the actual population to the target population is referred to as generalizability.

Analysis of registry outcomes first requires an analysis of recruitment and retention, of the completeness of data collection, and of data quality. Considerations include an evaluation of losses to followup; completeness for most, if not all, important covariates; and an understanding of how missing data were handled and reported. Analysis of a registry should provide information on the characteristics of the patient population, the exposures of interest, and the endpoints. Descriptive registry studies focus on describing frequency and patterns of various elements in a patient population, whereas analytical studies concentrate on associations between patients or treatment characteristics and health outcomes of interest. A statistical analysis plan describes the analytical plans and statistical techniques that will be used to evaluate the primary and secondary objectives specified in the study plan. Interpretation of registry data should be provided so that the conclusions can be understood in the appropriate context and any lessons from the registry can be applied to the target population and used to improve patient care and outcomes.

Evaluating Registries

Although registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions. The term "quality" can be applied to registries to describe the confidence that the design, conduct, and analysis of the registry can be shown to protect against bias and errors in inference—that is, erroneous conclusions drawn from the registry. Although there are limitations to any assessment of quality, a quality component analysis is used both to evaluate high-level factors that may affect results and to differentiate between research quality (which pertains to the scientific process) and evidence quality (which pertains to the data/ findings emanating from the research process). Quality components are classified as either "essential elements of good practice," which can be viewed as a checklist that should be considered for all patient registries, or as "potential enhancements to good practice," which may strengthen the value of the information in particular circumstances. The results of such an evaluation should be considered in the context of the disease area(s), the type of registry, and the purpose of the registry, and should also take into account feasibility and affordability.