SwissPedData: Standardising hospital records for the benefit of paediatric research

DOI: https://doi.org/10.4414/SMW.2021.w30069

Introduction

The creation of new evidence in medicine and the improvement of patient care are hampered by inefficient and laborious processes [1, 2]. Most evidence is gathered through stand-alone research projects that are costly, time-consuming, and conducted in an artificial research setting with a selected sample of patients. It also takes a long time for evidence to be implemented in health care [3]. Delays of many years are common, caused by the need to acquire research grants, recruit staff, obtain ethical approval, set up the study, recruit participants, collect and analyse data, write up and publish the results, and integrate these results into current standards of care. Paediatric research lags behind adult research for various reasons, including that the paediatric population is small, many paediatric health conditions are rare and ethical requirements are high. Given these constraints, results from studies in adults are often extrapolated to children [4, 5]. However, because of the important changes that occur during their development, children differ fundamentally from adults in many aspects. These include large age-related differences in susceptibility to environmental influences, in disease manifestations, in the adequacy and performance of diagnostic tests, in drug disposition, and in responses to treatment [6].

The digitalisation of health records could significantly improve the evidence for paediatric medicine and rare diseases as it potentially allows easy and fast access to clinical data from routine patient encounters. It could make clinical research faster and cheaper and make its results more representative of the patients typically seen in health care. Electronic health records (EHR) are widely used in hospitals to document clinical and administrative information about patient encounters. Unfortunately, EHR are rarely standardised within and between institutions and data are often entered into open text fields, resulting in unstructured data. Research on rare diseases relies on data from multiple centres and is limited by the time and costs required to extract and recode these data into a common format. Such data abstraction is particularly challenging when the original data are unstructured [7, 8]. Natural language processing and machine learning methods are increasingly being used to process unstructured data and make them available to research; however, many challenges remain [9]. Furthermore, retrospective standardisation often leads to a loss of information and impairment of data quality. These limitations could largely be circumvented if the original data were recorded in a structured and standardised way [10, 11]. A common EHR architecture allowing structured data capture during routine medical encounters could enable rapid analysis of healthcare data followed by speedy feedback of the knowledge generated into the same health care settings, a process called a learning health system [12, 13].

The aim of our project, which we have named SwissPedData, is to facilitate paediatric clinical research by improving and standardising the quality of data generated by paediatric health care in Switzerland. To achieve this, we first assessed the status quo, i.e., the relevant aspects of paediatric care for which data are collected, the way these data are recorded, and the data management systems used in the participating paediatric hospitals in Switzerland. Second, we developed and approved a standardised paediatric set of common data elements (CDEs) for EHR across Switzerland by conducting a multi-stage consensus finding process among general paediatricians and paediatric subspecialists of university and cantonal children’s hospitals. This paper describes the status quo of the project, the process of standardisation and the resulting set of CDEs: SwissPedData, Version 1.0.

Methods

SwissPedData taskforce

SwissPedNet, the research network of Swiss Children’s hospitals (https://www.swisspednet.ch/home/), received an infrastructure grant from the Swiss Personalized Health Network (SPHN) to develop a common data structure in paediatric hospitals and launched SwissPedData with the support of the Swiss Society of Paediatrics (https://www.paediatrieschweiz.ch). SPHN, an initiative of the Swiss Federal Government, aims to achieve a nationwide interoperability of health data produced in university hospitals (https://sphn.ch). SPHN funds the development of infrastructures that make health data shareable for research, following a decentralised approach where data remain in each hospital. Data sharing should become possible either through the direct transfer of individual health data or through distributed analyses, whereby the data do not travel, but are processed decentrally by algorithms and then only data summaries and results are transferred to a central location [14]. SwissPedData is coordinated by a taskforce that consists of a core team at the Institute of Social and Preventive Medicine, University of Bern (ISPM Bern) and representatives from all participating hospitals (fig. 1). All the university hospitals (Basel, Bern, Geneva, Lausanne and Zurich) and three cantonal children’s hospitals (Lucerne, St Gallen and Ticino) participated. The clinical directors of each hospital proposed one senior physician to represent the hospital’s management board and one junior physician to represent the house officers and registrars who enter the most data into the EHR. The directors also suggested senior physicians representing general paediatrics and all major paediatric subspecialties for collaboration as experts on the Delphi panel. Each hospital suggested at least one expert for general paediatrics and one for each subspecialty. These were then contacted by the core team. Distinct panels were set up for the following subspecialties: paediatric cardiology, endocrinology, gastroenterology, allergy/immunology, infectious diseases, metabolic diseases, nephrology, neurology, pulmonology and rheumatology. Paediatric oncology and neonatology were considered separately because standardised datasets for these subspecialties have already been developed by the Swiss Neonatal Network & Follow-Up Group (SwissNeoNet, https://www.neonet.ch/swissneonet) [15] and the Childhood Cancer Registry (https://www.childhoodcancerregistry.ch) [16]. Both datasets have been in use for many years and have been continuously refined and thus could be included directly in SwissPedData without further discussions. A related project is developing a set of CDEs for paediatric emergency medicine using the same approach. The results of that effort will be reported separately.

Figure 1 SwissPedData taskforce.

SwissPedData scope

SwissPedData focuses on the standardisation of the documentation of clinical encounters by paediatricians in children’s hospitals. This documentation encompasses medical history, physical examination, investigations, diagnosis, treatment and procedures. It excludes laboratory data and biospecimens, as these types of data are usually not entered into EHR by the clinicians themselves. Other SPHN-funded projects are working towards the harmonization of laboratory data in Switzerland (https://sphn.ch/fr/network/project-overview/).

Preparatory steps

To prepare the ground for determining the new set of CDEs, the core team assessed the current status of clinical data documentation during routine encounters in participating hospitals and in ongoing clinical registries and cohort studies. They then searched the literature for other initiatives aiming to standardise paediatric EHR (fig. 2). The core team visited each participating hospital and collected clinical data entry forms and information on the EHR system used and on the degree of digitalisation ­of health records. The team identified any large existing national or regional clinical paediatric registries and cohort studies via the registry centre (https://www.paediatrieschweiz.ch/swisspedregistry/) and the clinical hubs of SwissPedNet and through information obtained from the task force members of the participating hospitals. The core team collected metadata describing the datasets collected in these registries and cohort studies and investigated the content and format of the variables.

Figure 2 Consensus finding process followed to define SwissPedData, a set of CDEs for recording routine encounters in children’s clinics in Switzerland. CDE: Common Data Element

The core team also conducted a non-systematic, focused literature search to identify approaches to standardising paediatric data across multiple centres in other countries. The reference lists of the relevant publications identified were also scanned.

Selection of candidate common data elements for SwissPedData

Based on the information gained in the preparatory phase, the core team defined an initial list of CDEs to be considered for inclusion in the main module (general paediatrics) of SwissPedData. This was done based on an overview of the clinical data routinely documented in the hospitals; the variables collected in ongoing clinical cohort studies and registries; and the datasets of similar international initiatives. The initial list of CDEs was further refined during a two-day retreat held at the ISPM Bern with an interdisciplinary group including six paediatricians, three paediatric epidemiologists and two paediatric registry managers.

For each paediatric subspecialty, the initial list of candidate CDEs was drafted by the core team together with one hospital paediatrician who represented the subspecialty. This first draft was based on existing datasets specific to each subspecialty, such as large cohort studies or clinical registries, and/or on expert opinion (fig. 2, selection of candidate CDEs).

Reaching a consensus: the Delphi process

The consensus finding process aimed to reach agreement on 1) a list of CDEs for SwissPedData, 2) a standardised answer format for each CDE and 3) a classification of each CDE as either mandatory, recommended or optional. Starting with the initial selection of candidate CDEs, we implemented four Delphi rounds, consisting of one face-to-face meeting and three online surveys, to obtain a final set of CDEs based on a broad consensus (fig. 2). The Delphi method achieves consensus through a multi-round iterative process that involves eliciting opinions from experts and controlled feedback from the coordinating team [17, 18]. The same basic scheme was followed for the main general paediatric module and for each of the subspecialty modules. All experts were invited to each round, irrespective of whether or not they had given inputs in the previous rounds. For each online survey, the experts were asked to complete the questionnaire within two weeks. Those who had not responded within one week received a reminder e-mail. The online surveys were programmed with the software SurveyMonkey Inc., San Mateo, California, USA and analysed using Microsoft Excel.

In the first round, the experts evaluated the candidate CDEs according to their relevance for research and clinical work (fig. 2, round 1). Each expert was asked to vote for the inclusion or exclusion of each candidate CDE and to suggest any additional CDEs. The questions were: “please state for each of the proposed variables (CDEs) below whether you think they should be included in this subspecialty module of SwissPedData” and “would you add other variables (CDEs)?”. When opting for inclusion of a CDE, experts were further asked to classify the CDE as “mandatory”, “recommended” or “optional”. We retained CDEs that reached 80% for inclusion (designated as agreed) and excluded CDEs for which 80% of experts voted for exclusion. All other CDEs, including the additional CDEs suggested by the experts, were classified as “controversial”. There is no standard level of consensus in the literature, but levels ranging from 50% to 80% are commonly used [19, 20].

The second round consisted of face-to-face meetings, which were moderated by the core team and held at the ISPM Bern. During the face-to-face meetings, participants discussed all controversial CDEs and the additional CDEs suggested in the first online survey. They also agreed on standardised answer formats for the included CDEs. Eligible answer formats were a date, a date and time, a number, a binary response (e.g., yes/no), standardised response options or free text. When the discussions did not lead to a consensus, we used majority voting. Each face-to-face meeting lasted about three hours.

The third round was another e-Delphi survey, with participants being asked to check if key CDEs for their discipline were missing and to propose standardised answer formats or response options where these were missing.

In the fourth and final round, the agreed CDEs and answer formats were sent by email to all the experts for any last inputs and final approval.

Ethical approval was not required for this study, which did not involve the collection or use of patients’ data.

Results

Current status of EHR in participating hospitals and existing initiatives aiming to standardize paediatric data

The eight participating hospitals were using different clinical systems for EHR from various vendors (table 1). Their degree of digitalisation varied: while some hospitals were using EHR for all care processes, others were only doing so for some. For example, all hospitals were recording clinical notes relating to inpatients electronically, but only half of them were using electronic drug prescriptions at the time of the survey.

We identified 5 paediatric cohort studies and 25 paediatric clinical registries with a nationwide or multiregional reach (appendix 1). The focused literature search identified four projects with similar goals in other countries, namely PECARN (Pediatric Emergency Care Applied Research Network), PHIS+ (Pediatric Health Information System), PROS (Pediatric Research in Office Settings) and PEDSnet. The initiative most similar to ours was PEDSnet, an American national paediatric learning health system that was founded in 2014 by eight children’s hospitals, primarily to obtain child-specific data on the efficacy and safety of new and approved drugs [21] (https://pedsnet.org/data/). Currently, PEDSnet hosts analysis-ready, standardised longitudinal data from the primary, secondary and tertiary care of over 6.5 million patients. PEDSnet uses a common interoperable data platform that optimises the use of EHR, ensuring that data are entered once only. The collected data include demographics, vital status, encounters, diagnoses, vital signs, treatment and immunisations, among others (https://pedsnet.org/data/common-data-model/).

Consensus finding process (Delphi method)

Clinical directors proposed 121 experienced general paediatricians and subspecialists for the Delphi process, of whom 119 agreed to participate. Of these, 73 took part in the first round (online survey), 45 attended the second round (face-to-face meetings), 58 commented in the third round of the Delphi process and 68 gave their final approval of the dataset (appendix 2). The working groups contained between 7 and 14 members. All disagreements could be settled during the process through majority voting or through discussions. Most disagreements were about answer format rather than about which CDEs should be included in SwissPedData.

SwissPedData (Version 1.0)

SwissPedData consists of 336 CDEs: 76 in the main module on general paediatrics and between 11 and 59 in each of the 10 subspecialty modules (table 2 and appendix 3). The main module covers aspects concerning all paediatric patients, whether they are outpatients or inpatients. The subspecialty modules cover aspects specific to paediatric subspecialties that are not already covered by the main module. Each module is formally structured into the same nine domains representing all care processes: 1. Care Site, 2. Demographics, 3. Medical History, 4. Physical Examination, 5. Clinical Scores, 6. Investigations, 7. Diagnosis, 8. Treatment, and 9. Equipment and Procedures. These represent domains commonly covered by EHR. The Care Site domain contains administrative data related to the hospital and to patient encounters. It includes type of admission, length of stay and scheduled follow-up. The Demographics domain contains demographic data, for example date of birth, gender, address, and country of birth. The Medical History and Physical Examination domains include clinical information such as birth history, family history, symptoms, medications and vital signs. The Clinical Scores domain contains specific scores, for example triage scale for emergency department patients or developmental tests. The Investigations domain contains data on investigations performed, such as lung function, renal ultrasound or blood glucose monitoring for patients with diabetes. The Diagnosis domain includes diagnosis and date of diagnosis, as well as diagnosis classifications such as Online Mendelian Inheritance in Man (OMIM) codes. The Treatment domain contains data on medications prescribed and administered in hospital, treatment adverse events and reasons for discontinuation of treatment. The Equipment and Procedures domain contains data on procedures performed on the patient, such as dialysis.

The full set of CDEs is shown in appendix 3, which provides a complete list of all agreed CDEs along with their description, answer format and standardised response options, and importance (mandatory, recommended or optional). Answer choices are number, binary or standardised options, or free text. When the “standardised option” format is used, specific value sets are defined. The CDEs will be implemented in children’s hospital EHR depending on their importance, categorised as mandatory, recommended or optional. Mandatory CDEs must be implemented in EHR by all participating hospitals. Recommended CDEs should be implemented and optional CDEs may be implemented at the discretion of each hospital.

Examples of mandatory CDEs are vital parameters in the main module (general paediatrics) or “route of feeding” in the gastroenterology module. In the latter case, “route of feeding” will be recorded with standardised response options (oral, gastrostomy, naso/orogastric tube, intravenous, other). An example of a recommended CDE is “seizure type according to the ILEA 2017 classification of seizures” in the neurology module. “Opening pressure at lumbar puncture” is an optional CDE in the same module (appendix 3). 

Discussion

We developed SwissPedData, a standardised national set of CDEs designed to collect clinical data during paediatric routine encounters in a harmonised way. It is the result of a broad consensus between general paediatricians and paediatric subspecialists from eight university and cantonal children’s hospitals in Switzerland. It describes all processes of paediatric medical care including clinical and paraclinical assessment, diagnosis, treatment, disposition and care site. Each part of the dataset follows the usual structure of the EHR to allow easy implementation.

Clinical data standardisation for a Swiss paediatric learning healthcare system

SwissPedData aimed to standardise items up-front at the point of data entry. Prospective, standardised recording of routine clinical encounters avoids duplicate entry into research databases. However, this should not happen at the expense of an increase in documentation time by clinicians, a concern raised during our Delphi process. To avoid this pitfall, we focused primarily on data elements that are not only useful for research, but also for clinical work, and included CDEs that are routinely documented in paediatric EHR. SwissPedData is not comprehensive and much of the clinical documentation will remain unstandardised to preserve the rich narrative details that are difficult to capture in standardised fields but are nevertheless important for daily clinical work. These narrative data could be used by researchers applying text-mining approaches. SwissPedData could also be supplemented by questionnaires to patients and their families. The implementation of SwissPedData in EHR will include careful attention to clinician workflow to minimise potential negative consequences of standardisation.

SwissPedData is designed to provide a basis for a paediatric learning health system in Switzerland in which clinical data from different children’s hospitals can be combined to rapidly generate new knowledge relevant for day-to-day practice and translate it into improved health care for children. Existing learning health systems in other countries, such as PEDSnet in the US, have demonstrated that a paediatric learning health system can improve the health outcomes of children [22, 23]. Examples include the rapid identification of children suffering from glomerular diseases for clinical trials [24], comparing weight loss and safety among bariatric procedures using EHR data [25] and, recently, describing the epidemiology of paediatric patients infected by SARS-CoV-2 [26].

Strengths and limitations

The main strength of SwissPedData is that it is based on broad agreement between paediatricians from all university and cantonal paediatric clinics in Switzerland. The project received strong support from all clinical directors of Swiss children’s hospitals, from the paediatric research network SwissPedNet and from more than 100 experienced paediatricians who participated in its development. SwissPedData emphasises the prospective collection of standardised data, which can greatly reduce the time and costs needed for data preparation and analysis as it avoids the need for retrospective standardisation or double entry. Our consensus finding approach could be adapted for use by other medical specialties that wish to define CDEs in the future.

SwissPedData has a number of omissions that are intentional. First, we focused on standardising a minimal set of items that are particularly relevant and specific to paediatric routine care. SwissPedData will thus not replace existing terminologies for clinical health care such as SNOMED-CT. Rather, standardised data from SwissPedData can in the future be mapped to SNOMED-CT. Second, SwissPedData does not include laboratory data or detailed radiological data. However, other projects within the SPHN are working on the standardisation of these domains. The goal is to link the standardised paediatric data extracted from EHR with laboratory data standardised thanks to other SPHN projects like L4CHLAB. Such linkage can be done through hospital patient IDs, or with birth dates and names. Third, SwissPedData will need to be translated into the Swiss national languages before implementation in children’s hospital EHR.

SwissPedData is adapted to the Swiss context

The Swiss healthcare system is decentrally structured, with cantons being responsible for the organisation of local health care, and therefore is highly heterogeneous. As a consequence, children’s clinics are relatively small, with catchment areas of a few 100,000 children. Obtaining sufficient patient samples for research is only possible by combining data from multiple hospitals, especially for rare conditions. However, given the differences in EHR and IT systems between hospitals, this results in long delays and huge costs for obtaining, extracting, standardising and cleaning the heterogeneous data. SwissPedData, once implemented in all children’s clinics, will allow researchers to identify and recruit patients for clinical trials in real time, to conduct retrospective studies with high-quality data, and to conduct nested prospective studies. As examples, participants of the “Clinical Data for Paediatric Research: the Swiss Approach” symposium held in 2019 drafted sketches of the following research projects based on SwissPedData: a diagnostic study on the validity of the tests used for auditory screening in newborns; a benchmarking study assessing the quality of treatment for bronchiolitis across different children’s hospitals; a cohort study on the incidence of hearing loss after treatment with aminoglycosides in infancy; a cohort study on kidney injury after treatment with acyclovir; and a randomised clinical trial comparing the effectiveness of different treatment regimens for type 1 diabetes. Some of these project sketches suggested complementing the hospital dataset with available data from other sources such as the federal statistical office or laboratory data, or through the collection of additional data through questionnaires or specific examinations.

Comparison with other projects

SwissPedData is closely aligned with PEDSnet, a US-based paediatric clinical data research network [21]. PEDSnet includes eight children’s hospitals that provide care for 2.8% of the paediatric population in the USA (2.1 million patients) [21]. The database contains standardised clinical data from EHR covering 6.5 million children (https://pedsnet.org/) and forms the basis of a high-quality research programme and learning health system. Studies based on PEDSnet data cover a wide range of research topics and study designs in paediatrics, including descriptive epidemiology [27], computable phenotyping [24], longitudinal observational studies [28] and comparative effectiveness [29]. PEDSnet established a common data model (PEDSnet CDM) from the beginning of their network, based on the Observational Health Data Sciences and Informatics collaborative's OMOP common data model. With SwissPedData, we defined a list of priority CDEs that can be mapped to SNOMED-CT in the future.

PEDSnet may also serve as a role model for the implementation of SwissPedData and has already demonstrated its usefulness for observational and interventional research and for the standardisation of care processes. Each hospital that participates in PEDSnet regularly extracts the standardised data from its EHR in a predefined way [21].

Another notable example of harmonised clinical datasets in paediatrics is the Pediatric Emergency Care Applied Research Network (PECARN), an EHR-based registry that has harmonised data in the paediatric emergency setting in seven American paediatric emergency departments to make it usable for paediatric research. PECARN uses data resources from seven paediatric emergency departments of four hospitals [30].

Outlook and next steps

All participating hospitals are committed to implementing SwissPedData in their EHR by 2024. A committee of clinicians and IT specialists in each hospital will supervise the implementation process. The EHR of children’s hospitals will be restructured at the front-end to include SwissPedData CDEs. Practically, this means that EHR as seen by their users (physicians) will include the CDEs of SwissPedData. For some hospitals, where this is not possible in the short term, we will also offer the possibility of transforming the source data to the CDEs and contributing it to the common dataset. SwissPedData is intended to evolve and be adaptive to existing needs. The set of CDEs can be expanded to cover more domains or to include more CDEs per domain. Temporary CDEs can be added for nested research projects. Self-completed or parent-completed questionnaires can add information relating to a child’s family and home environment, which is not routinely recorded in EHR. Data from primary care encounters could also be integrated in the future.

In ongoing work, other prerequisites for the implementation of SwissPedData are being put into place: a general consent form for use of the data from patients and caregivers, a data transfer and use agreement (DTUA) between the clinics, and protocols for obtaining ethics approval for SwissPedData overall and for individual research projects. Some aspects are being dealt with within other infrastructure development projects of the SPHN network (www.sphn.ch), namely the C3-Study (citizen centred consent) project and the E-General Consent project. Furthermore, the SPHN provides legal agreement templates, including a DTUA and an ethical framework for all its projects. It is important to stress that only data useful for the clinical management of the patient will be recorded and that these data will always be stored by each children’s hospital as part of the patient’s file. The only difference to the previous procedure is that some of these clinical data will be recorded in a standardised way. To have access to these data for research, researchers will have to get ethical approval as usual.

It is planned that SwissPedData will be implemented as a project on the SPHN infrastructure for data exchange, so that data can in future be accessed through a central portal. The SPHN Data Coordination Centre and BioMedIT (https://sphn.ch/network/projects/biomedit/) can provide assistance and the infrastructure for this. The aim is to keep SwissPedData CDEs harmonized with the future releases of the SPHN dataset (https://sphn.ch/services/documents/technical-documents/). An additional central coordination center for paediatric research should facilitate communication between children’s clinics, international research partners and funders, and also assist researchers in writing grant applications, obtaining ethical approval and accessing the necessary datasets. The resources needed to maintain SwissPedData will require the support of a central coordination center encompassing an experienced researcher ideally with a background in paediatrics, an IT specialist, andlocal support of the responsible clinicians and IT specialists in each hospital. Funding for the implementation and maintenance of SwissPedData will need to be secured. Potential funding sources are participation in suitable calls for proposals, charging cost-covering fees for services provided by SwissPedData and collaboration with industry, for example for post-marketing studies. Collaborations with international partners such as PEDSnet are foreseen, and first exchanges have occurred.

In conclusion, SwissPedData defines a set of common data elements (CDEs) for clinical paediatric care based on a broad agreement among university and cantonal paediatric hospitals in Switzerland. With SwissPedData, Swiss children’s hospitals will be able to provide researchers with standardized, high-quality routine clinical paediatric data in the near future. SwissPedData will provide the basis for a learning health system for paediatric care in Switzerland.

Acknowledgments

We thank all the experts who participated in the Delphi process, SwissPedNet, College A, the Swiss Personalized Health Network (SPHN), and ISPM Bern staff: Alexander Laemmle, Alexander Moeller, Alexandra Wilhelm-Bals, Alexandre Datta, Alice Koehli, Andrea Duppenthaler, Andreas Nydegger, Andreas Worner, Anita Rauch, Anna Wefers, Anne Tscherter, Arnaud Merglen, Barbara Goeggel Simonetti, Juerg Barben, Birgit Donner, Caroline Roduit, Christian Braegger, Christian Kahlert, Christian Korff, Christian Huemer, Christian Lovis, Christina Schindera, Christoph Aebi, Christoph Berger, Christophe Folly, Christoph Rudin, Christian Balmer, Cristina Ardura, Claudia Boettcher, Constance Barazzone-Argiroffo, Corinna Leoni Foglia, Dagmar L’Allemand, Daniel Konrad, Daniel Trachsel, Daniela Marx-Berger, Diana Ballhausen, Dirk Fischer, Dominik Stambach, Eliane Roulet, Elvira Cannizzaro, Emanuela Valsangiacomo, Eva Pedersen, Federica Vanoni, Felicitas Bellutti, Florian Bauder, Florence Barbey, Florian Singer, François Cachat, Franziska Kunz, Gabor Szinnai, Georg Marx, Giovanni Ferrari, Gianluca Gualco, Guido Laube-Bless, Hans Peter Kuen, Hassib Chehade, Ilse Kern, Isabel Bolt, Isabelle Rochat, Jana Pachlopnik Schmid, Jean-Baptiste Armengaud, Jean-Christoph Caubet, Joan Carles Suris Granell, Joël Fluss, Johannes Spalinger, Julien Caccia, Jürg Hammer, Kanetee Busiah, Katrin Heldt, Katharina Flandera, Kristina Keitel, Laetitia Marie Petit, Lisa Kottanattu, Lorenzo Zgraggen, Luca Garzoni, Matthias Horn, Maria Otth, Matthias Baumgartner, Matthias Gautschi, Maura Zanolari-Calderari, Maurice Beghetti, Melanie Hess, Michael Hauschild, Michael Buettcher, Michael Hofer, Mirjam Dirlewanger, Myrofora Goutaki, Nicolas Regamey, Nicolas Waespe, Nicole Sekarski, Nicole Ritz, Noémie Wagner, Oliver Niesse, Oswald Hasselmann, Paloma Parvex, Paolo Tonella, Paolo Paioni , Pascale Wenger, Peter Weber, Philip Broser, Philipp Agyman, Philip Do Canto, Philippe Steenhout, Philippe Eigenmann, Piero Balice, Pierre-Alex Crisinel, Raoul Furlano, Rebeca Mozun, Regula Laux, Regina Wespi, Robert Steinfeld, Sabine Pallivathukal, Sandra Asner, Sebastian Grunt, Sébastien Lebon, Sébastien Papis, Selina Pinosch, Sibylle Tschumi, Stefano di Bernardo, Sylvain Blanchon, Thomas Schmitt-Mechelke, Ulrike Halbsguth, Urs Zumsteg, Valérie Schwitzgebel, Valérie McLin, Verena Pfeiffer and Yacine Aggoun.

Appendices

Tables S1-S3 are available as appendices in the PDF version of the manuscript.

References

1. Califf RM , Robb MA , Bindman AB , Briggs JP , Collins FS , Conway PH , et al.  Transforming Evidence Generation to Support Health and Health Care Decisions. N Engl J Med. 2016 Dec;375(24):2395–400. https://doi.org/10.1056/NEJMsb1610128  

2. Khozin S , Blumenthal GM , Pazdur R ; Real-world Data for Clinical Evidence Generation in Oncology . Real-world Data for Clinical Evidence Generation in Oncology. J Natl Cancer Inst. 2017 Nov;109(11). https://doi.org/10.1093/jnci/djx187  

3. Wensing M , Grol R . Knowledge translation in health: how implementation science could contribute more. BMC Med. 2019 May;17(1):88. https://doi.org/10.1186/s12916-019-1322-9  

4. Kern SE . Challenges in conducting clinical trials in children: approaches for improving performance. Expert Rev Clin Pharmacol. 2009 Nov;2(6):609–17. https://doi.org/10.1586/ecp.09.40  

5. The Lancet Diabetes Endocrinology . Spotlight on rare diseases. Lancet Diabetes Endocrinol. 2019 Feb;7(2):75. https://doi.org/10.1016/S2213-8587(19)30006-3  

6. Davis MM . Stunting the growth of child health research: a need to reframe “children are not small adults”. JAMA Pediatr. 2013 Jul;167(7):598–9. https://doi.org/10.1001/jamapediatrics.2013.165  

7. Polnaszek B , Gilmore-Bykovskyi A , Hovanes M , Roiland R , Ferguson P , Brown R , et al.  Overcoming the Challenges of Unstructured Data in Multisite, Electronic Medical Record-based Abstraction. Med Care. 2016 Oct;54(10):e65–72. https://doi.org/10.1097/MLR.0000000000000108  

8. Ogunyemi OI , Meeker D , Kim HE , Ashish N , Farzaneh S , Boxwala A . Identifying appropriate reference data models for comparative effectiveness research (CER) studies based on data from clinical information systems. Med Care. 2013 Aug;51(8 Suppl 3):S45–52. https://doi.org/10.1097/MLR.0b013e31829b1e0b  

9. Tayefi M , et al. Challenges and opportunities beyond structured data in analysis of electronic health records. WIREs Computational Statistics. n/a(n/a): p. e1549. 

10. Kush R , Alschuler L , Ruggeri R , Cassells S , Gupta N , Bain L , et al.  Implementing Single Source: the STARBRITE proof-of-concept study. J Am Med Inform Assoc. 2007 Sep-Oct;14(5):662–73. https://doi.org/10.1197/jamia.M2157  

11. Breil B , Semjonow A , Müller-Tidow C , Fritz F , Dugas M . HIS-based Kaplan-Meier plots—a single source approach for documenting and reusing routine survival information. BMC Med Inform Decis Mak. 2011 Feb;11(1):11. https://doi.org/10.1186/1472-6947-11-11  

12. Seid M , Margolis PA , Opipari-Arrigan L . Engagement, peer production, and the learning healthcare system. JAMA Pediatr. 2014 Mar;168(3):201–2. https://doi.org/10.1001/jamapediatrics.2013.5063  

13. Institute of Medicine Roundtable on Evidence-Based . M., The National Academies Collection: Reports funded by National Institutes of Health, in The Learning Healthcare System: Workshop Summary, L. Olsen, D. Aisner, and J.M. McGinnis, Editors. 2007, National Academies Press (US) 

14. Toh S , Rifas-Shiman SL , Lin PD , Bailey LC , Forrest CB , Horgan CE , et al.; PCORnet Antibiotics and Childhood Growth Study Group . Privacy-protecting multivariable-adjusted distributed regression analysis for multi-center pediatric study. Pediatr Res. 2020 May;87(6):1086–92. https://doi.org/10.1038/s41390-019-0596-0  

15. Adams M , Bucher HU . Neonatologie: Ein früher Start ins Leben: Was bringt ein nationales Register? Schweiz Med Forum. 2013;13(3):35–7. https://doi.org/10.4414/smf.2013.01397  

16. Michel G , von der Weid NX , Zwahlen M , Adam M , Rebholz CE , Kuehni CE ; Swiss Childhood Cancer Registry; Swiss Paediatric Oncology Group (SPOG) Scientific Committee . The Swiss Childhood Cancer Registry: rationale, organisation and results for the years 2001-2005. Swiss Med Wkly. 2007 Sep;137(35-36):502–9.  

17. Jones J , Hunter D . Consensus methods for medical and health services research. BMJ. 1995 Aug;311(7001):376–80. https://doi.org/10.1136/bmj.311.7001.376  

18. Dalkey NC . The Delphi method: An experimental study of group opinion. 1969, RAND CORP SANTA MONICA CALIF. 

19. Hasson F , Keeney S , McKenna H . Research guidelines for the Delphi survey technique. J Adv Nurs. 2000 Oct;32(4):1008–15.  

20. Walker MA , Selfe MJ . The Delphi method: a useful tool for the allied health researcher. Br J Ther Rehabil. 1996;3(12):677–81. https://doi.org/10.12968/bjtr.1996.3.12.14731  

21. Forrest CB , Margolis PA , Bailey LC , Marsolo K , Del Beccaro MA , Finkelstein JA , et al.  PEDSnet: a National Pediatric Learning Health System. J Am Med Inform Assoc. 2014 Jul-Aug;21(4):602–6. https://doi.org/10.1136/amiajnl-2014-002743  

22. Ramsey LB , Mizuno T , Vinks AA , Margolis PA . Learning Health Systems as Facilitators of Precision Medicine. Clin Pharmacol Ther. 2017 Mar;101(3):359–67. https://doi.org/10.1002/cpt.594  

23. Deans KJ , Sabihi S , Forrest CB . Learning health systems. Semin Pediatr Surg. 2018 Dec;27(6):375–8. https://doi.org/10.1053/j.sempedsurg.2018.10.005  

24. Denburg MR , Razzaghi H , Bailey LC , Soranno DE , Pollack AH , Dharnidharka VR , et al.; Using Electronic Health Record Data to Rapidly Identify Children with Glomerular Disease for Clinical Research . Using Electronic Health Record Data to Rapidly Identify Children with Glomerular Disease for Clinical Research. J Am Soc Nephrol. 2019 Dec;30(12):2427–35. https://doi.org/10.1681/ASN.2019040365  

25. Arterburn D , Wellman R , Emiliano A , Smith SR , Odegaard AO , Murali S , et al.; PCORnet Bariatric Study Collaborative . Comparative Effectiveness and Safety of Bariatric Procedures for Weight Loss: A PCORnet Cohort Study. Ann Intern Med. 2018 Dec;169(11):741–50. https://doi.org/10.7326/M17-2786  

26. Bailey LC , Razzaghi H , Burrows EK , Bunnell HT , Camacho PE , Christakis DA , et al.  Assessment of 135 794 Pediatric Patients Tested for Severe Acute Respiratory Syndrome Coronavirus 2 Across the United States. JAMA Pediatr. 2021 Feb;175(2):176–84. https://doi.org/10.1001/jamapediatrics.2020.5052  

27. Bailey LC , Milov DE , Kelleher K , Kahn MG , Del Beccaro M , Yu F , et al.  Multi-Institutional Sharing of Electronic Health Record Data to Assess Childhood Obesity. PLoS One. 2013 Jun;8(6):e66192–66192. https://doi.org/10.1371/journal.pone.0066192  

28. Lang JE , Bunnell HT , Hossain MJ , Wysocki T , Lima JJ , Finkel TH , et al.  Being Overweight or Obese and the Development of Asthma. Pediatrics. 2018 Dec;142(6):e20182119. https://doi.org/10.1542/peds.2018-2119  

29. Inge TH , Coley RY , Bazzano LA , Xanthakos SA , McTigue K , Arterburn D , et al.; PCORnet Bariatric Study Collaborative . Comparative effectiveness of bariatric procedures among adolescents: the PCORnet bariatric study. Surg Obes Relat Dis. 2018 Sep;14(9):1374–86. https://doi.org/10.1016/j.soard.2018.04.002  

30. Deakyne Davies SJ , Grundmeier RW , Campos DA , Hayes KL , Bell J , Alessandrini EA , et al.; Pediatric Emergency Care Applied Research Network . The Pediatric Emergency Care Applied Research Network Registry: A Multicenter Electronic Health Record Registry of Pediatric Emergency Care. Appl Clin Inform. 2018 Apr;9(2):366–76. https://doi.org/10.1055/s-0038-1651496