a Faculty of Law, University of Zurich, Switzerland
b Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Switzerland
c Institute for Complementary and Integrative Medicine, University Hospital Zurich, Switzerland
d Department of Geography, University of Zurich, Switzerland
e Institute for Implementation Science in Health Care, University of Zurich, Switzerland
As COVID-19 spreads across the globe, crowdsourced digital technology harbours the potential to improve surveillance and epidemic control, primarily through increased information coverage, higher information speed, fast case tracking and improved proximity tracing. Targeting those aims, COVID-19-related smartphone and web-based health applications are continuously emerging, leading to a multitude of options, raising ethical and legal challenges and potentially overwhelming end users.
Building on an existing trustworthiness checklist for digital health applications, we searched the literature and developed a framework to guide the assessment of smartphone and web-based applications that aim to contribute to controlling the current epidemic or mitigating its effects. It further integrates epidemiological subject knowledge and a legal analysis, outlining the mechanisms through which new applications can support the fight against COVID-19.
The resulting framework includes 40 questions across 8 domains on “purpose”, “usability”, “information accuracy”, “organisational attributes / reputation”, “transparency”, “privacy” and “user control / self-determination”. All questions should be primarily answerable from publicly available data, as provided by application manufacturers. The framework aims to guide end users in choosing a transparent, safe and valuable application and suggests a set of information items that developers ideally make available to allow a balanced judgement and facilitate the trustworthiness of their products.
Keywords: digital health, mHealth, COVID-19, evaluation, checklist, epidemiology, law
The SARS-CoV-2 (COVID-19) pandemic is barely 6 months old but has swept across countries and continents with an almost unprecedented speed (e.g., as tracked in ). The fast transmission of SARS-CoV-2 also exposed weaknesses in current monitoring practices in Switzerland, which heavily rely on decentralised reporting of positive laboratory tests, flowing from diagnosing clinicians to cantonal and federal health authorities, as well as on the count of general practitioner consultations due to flu-like illnesses (“Sentinella System”) .
The ability to take timely action, however, depends on early warning systems that provide nearly real-time, actionable information on emerging diseases. Crowdsourced digital tools, relying on citizens to collect data, offer promising possibilities for faster, population-based monitoring of symptoms and movement patterns [3, 4]. In fact, several crowdsourced smartphone and web-based health applications (henceforth referred to as apps) have emerged since the start of the COVID-19 pandemic and the beginning of the lockdown. Many of these have different purposes (e.g., monitoring of the epidemic, tracing, or tracking of own movements) and quite distinct technical approaches (e.g., the amount of collected personal data, centralised vs decentralised storage of data) . The common element of many of these apps is that citizens actively or passively (by the use of smartphone sensors) collect and store certain data that, when aggregated, can be informative about the current state of the COVID-19 epidemic. One example in Switzerland is covidtracker.ch, which collects and maps flu-like symptom reports. Other apps may also passively collect sensor data, such as geolocation or Bluetooth-enabled tracking and proximity tracing.
The multitude of technologies and approaches of emerging apps in the context of COVID-19 also harbour risks for end users and potential for confusion. From a privacy perspective, the collection of data through digital devices is extremely sensitive, for which robust privacy safeguards are of paramount importance . For example, the collection of geolocation data poses special challenges for privacy, depending on the accuracy of geolocation positioning and of proximal distance measurements [7, 8]. Furthermore, as data are often collected under the premise of very broad goals, (e.g., their anonymised use by health authorities to predict the future course of the epidemic), non-specialists can rarely judge whether these are achievable and truly contributing towards a better epidemic control. Along the same lines, app descriptions rarely provide enough information to allow end users to assess the risk for privacy rights infringement or data ownership loss. Ultimately, these uncertainties cause confusion, which in turn may lead to low acceptance and use. Low participation is a real risk for potentially reliable and privacy-preserving software, undermining the well-intended goals of many crowdsourced apps for better epidemic monitoring and tracking.
This study aimed to develop a framework to provide guidance on the alignment of a specific app with epidemiological principles to contain the epidemic spread, as well as on compliance with the current legal basis in Switzerland concerning privacy and constitutional rights. Based on an existing framework , we synthesised domains that facilitate the assessment of an app’s trustworthiness, enriched by technical, epidemiological and legal aspects. The incorporation of these aspects supports crowdsourced apps to effectively achieve their purpose (such as to help managing the epidemic or its consequences on personal well-being) while also adhering to the individual rights.
Characteristics of the COVID-19 pandemic and current monitoring/surveillance weaknesses
In order to understand the role of apps in controlling the COVID-19 pandemic, it is key to consider viral and epidemiological characteristics, as well as current weaknesses in disease monitoring/surveillance.
The current COVID-19 pandemic shares some similarities with seasonal influenza. First, the ongoing epidemic phase of COVID-19 in Switzerland coincides with the regular influenza season, and symptoms overlap [10, 11]. Both infections can trigger symptoms such as fever, dry cough, general malaise, headaches, sneezing or muscle aches. A second commonality is that both the influenza virus and SARS-CoV-2 can be effectively transmitted while a person is still pre-symptomatic. Nevertheless, some differences in symptomologies have started to emerge. For example, loss of smell or taste was relatively frequently reported in persons with COVID-19 and is now considered a distinguishing symptom .
However, the COVID-19 pandemic is in many ways more dangerous than seasonal influenza. Not only does SARS-CoV-2 spread faster than a seasonal influenza , it also seems to be associated with a higher case fatality ratio and a considerably higher proportion of cases needing intensive care and ventilation . Current data indicate that 80% of infections have a “mild” disease course, with the definition of “mild” disease course ranging from almost no symptoms to more severe symptoms and pneumonia, but not requiring ventilation . Of these, around 20% are estimated to remain asymptomatic. In persons who eventually develop symptoms, the median duration from infection to first symptom occurrence lies between 4 and 5 days, but can extend up to 14 days [14, 15]. Nevertheless, infectivity is believed to start 1–3 days before symptom onset , and transmission from pre-symptomatic persons is easily possible, as demonstrated by a case series from Bavaria . Indeed, some studies estimate the proportion of pre-symptomatic transmissions to be in the order of 40% [18, 19]. This is not surprising given that viral shedding primarily occurs in the upper respiratory tract with high viral loads that tend to peak at the onset of symptoms [16, 20]. These characteristics render transmission by droplets or fomites quite simple and are possible explanations for the high basic reproductive number (R0) of between 2 and 3 in uncontained epidemics (that is, a single infected person infects on average two to three other persons), thus resulting in an exponential increase of case numbers [21, 22].
From a public health and health authority viewpoint, the above-mentioned characteristics of the epidemic exposed some challenges in current surveillance practices. The main data source for surveillance is positive polymerase chain reaction (PCR) tests to measure Sars-CoV-2 viral loads, the results of which are reported to cantonal and federal authorities. However, the rapid dynamics of transmission, with an infectivity window of 1–3 days prior to symptom onset , as well as the estimated proportion of 10–20% of asymptomatic persons , suggest that daily reported case numbers capture only a fraction of the real number of new cases (incidence). Furthermore, because testing capacities were and remain limited, most countries have imposed more or less restrictive conditions as testing criteria for SARS-CoV-2, for instance by only testing persons with symptoms or specific risk exposures (e.g., contact with a confirmed SARS-CoV-2 positive person). Additionally, current reporting occurs around 1–2 days after sample collection and testing. Because of the epidemic’s transmission dynamics, this reporting delay is significant and may impede epidemic control and efforts to break transmission chains . The same applies to currently practiced methods of contact tracing, which rely on interviews of infected persons by health authorities and attempts to inform contacts with potential risk exposures by telephone calls .
The possible role of digital tools in personal management, care, and epidemic control of COVID-19
According to recommendations of the World Health Organization (WHO), “test, trace, isolate, quarantine” are cornerstones in the attempt to mitigate and later contain the COVID-19 pandemic . Digital tools can greatly support these measures in at least two ways. First, improved digital data flows, proximity tracing and geolocation tracking can speed up processes of reporting and contact tracing. For example, a recent modelling study illustrated that effective epidemic control strongly depends on the effective time between risk exposure, contact tracing and self-quarantine, and that even a one-day reduction can have a significant impact on epidemic control, provided that traced individuals enter self-quarantine immediately . Therefore, many countries consider apps that facilitate contact tracing as a possible element of a post-epidemic containment strategy.
Second, given the ubiquity of internet-connected devices, crowdsourced surveillance could markedly broaden the population base and increase the speed of surveillance to almost real-time. Indeed, internet-enabled citizen reports of flu-like illnesses have been around almost since the dawn of the internet (e.g., pioneered by GoogleFlu) and operationalised in several countries with some success [25, 26]. However, due to the temporal overlap with the influenza season and the shared similarities of symptoms, such crowdsourced reports are not very specific for COVID-19 (leading to a relatively large number of false positive reports). Nevertheless, a retrospective analysis from China illustrates that flu-like illness reports were indeed providing relatively marked early signals of the initial COVID-19 epidemic in Wuhan .
Table 1 lists additional examples of apps and platforms that are currently being published in the context of the COVID-19 pandemic, namely patient diaries, telehealth solutions and exposure risk assessments, as well as tools and interventions to mitigate the consequences of COVID-19 on mental health. Digital contact tracing and crowdsourced surveillance primarily aim to contribute to better epidemic control and therefore provide only limited personal benefit. Also, the intended effect of these apps is only achievable if a large fraction of persons in a given geographic area are actively utilising the tool. That is, there is a strong “density-dependence” of app usage and broader benefits.
|Description||Aim for societal |
|Aim for personal |
|Proximity tracing apps|
|Technology to store close contacts over a certain period; may include an automated warning system of contacts in case of COVID-19 diagnosis||X||X|
|Platform to monitor symptom occurrence on population level||X||X|
|Keeping track of symptoms and exposure risks (sometimes combined with population-based symptom surveillance)||(X)||X||(X)|
|Advisory platforms and self-assessments in case of symptoms, online consultation with medical experts||X|
|Mental health interventions|
|Technologies for behavioural interventions, alleviating fears, improvement of mental health||X|
In contrast to the two formerly described types of technologies (digital proximity tracing and crowdsourced monitoring), users can receive a personal benefit from some apps listed in table 1. Nevertheless, these can also contribute towards a more general, population level goal (e.g., aggregating diary data for disease surveillance purposes). The potential contributions of smartphone and web-based apps to epidemic control mechanisms are listed in table 2.
|Possible contributions of apps to epidemic management||Mechanism|
|Providing data from previously untested persons||Yields better estimate of number of affected persons|
|Performing a more timely monitoring of the epidemic||Provides case numbers faster, allows faster decision making|
|Accelerating processes (e.g., contact tracing)||Faster identification of contacts allows more complete and faster warning of potentially exposed persons|
|Discovering new aspects of dealing with the epidemic on a daily level||Personal reports help to investigate, for example, unwanted effects of epidemic control measures such as lockdowns|
|Collecting personal history (e.g. regarding exposure risks / symptoms)||Stored data help to assess possible risk exposures|
In sum, there is an increasing diversity of publicly available COVID-19 related smartphone and web-based apps, many with quite different objectives. This abundance can be very confusing and ultimately harm efforts that require a strong density-dependence and rely on a large and ideally representative participation of the target population. Therefore, now more than ever, assessing these emerging solutions and choosing the right ones should be based on adequate, reliable and easily accessible information.
Given potential consequences of smartphone and web-based app usage for end user privacy (particularly for apps relying on contact tracing or tracking technologies) an assessment of the Swiss legal context is crucial.
Compared with the US, Switzerland has strict privacy rights rules . The right to privacy is a fundamental right that includes the right of every person to be protected against the misuse of their personal data (Article 13 Federal Constitution of the Swiss Confederation). Multiple provisions in the Federal Act on Data Protection (FADP), as well as other regulations, specify and bolster this fundamental right.
The FADP provides an overall framework (not limited to health data) and deals with data protection using principles similar to those applied in other countries. The law has a wide scope and applies to personal data processed by Federal authorities, private organisations and individual persons.  “Personal data” protected by the FADP is defined as all information relating to an identified or identifiable person (Art. 3 lit. a FADP). Non-identifiable data does not fall within the scope of the FADP. Data are considered as non-identifiable only if an unreasonable technical effort leads to the re-identification of the individual. However, under Swiss law there is no specific definition of what an “unreasonable effort” means, which leads to uncertainties . Whether processed data must be qualified as “personal data” must be determined on a case-by-case basis. The Federal Data Protection and Information Commissioner supervises compliance of federal bodies with this Act. The Commissioner investigates cases either on their own initiative or at the request of a third party (Art. 27 FADP). With regard to private manufacturers and individuals it is, in general, the self-responsibility of manufacturers and individuals to ensure that the processed data comply with the FADP. On complaint, private persons are liable to a fine if they do not comply with the FADP (Art. 34 FADP). With regard to the decentralised privacy-preserving proximity tracing app (DP-3T app; see table 1), the Federal Data Protection and Information Commissioner actively assessed whether the app complies with the FADP .
Some relevant provisions of the FADP can be outlined as follows: Personal data may only be processed lawfully and for the purpose indicated at the time of collection, that is evident from the circumstances, or that is provided for by law. The collection of personal data and in particular the purpose of its processing must be evident to the data subject (Art. 4, FADP ). No more data than necessary may be processed. Furthermore, personal data must be protected against unauthorised processing through adequate technical and organisational measures (Art. 7, FADP ), e.g., the software has to align with the current state of the art and must be protected against risks, such as unauthorised or accidental destruction, technical faults, unlawful use, unauthorised access or other unauthorised processing. To avoid misuse and discrimination and to safeguard the individual’s privacy rights, personal data must be anonymised, ideally, at the first possible moment .
“Health data” according to the FADP are considered as “sensitive personal data” and require additional privacy safeguard measures to those outlined above for other personal data (Art. 3 FADP). For example, it is an additional legal requirement, that each user gives his or her “informed consent” after adequate information, i.e., voluntarily and expressly agrees to the processing of his or her data (Art. 4, FADP ). Furthermore, each individual has the right to opt-out anytime and may demand the deletion of their data. “Health data” are defined as data that allow direct or indirect conclusions regarding the physical or mental health status of an individual [33, 34]. Although there is no doubt that symptom data qualify as health data, tracking or tracing data (also defined as “mobility data”) may not seem to qualify as health data at first glance. However, since such data give indirect conclusions about the physical health status of an individual, they also qualify as “health data”, i.e., “sensitive personal data”. 
To better safeguard individual privacy rights and to prevent misuse and discrimination as much as possible, additional legal and ethical prerequisites have been developed [36, 37]. For example, suggestions are that data should be stored decentralised, that the government should play a leading role, and that the government, i.e., the Federal Council, should enact the basic principles for the implementation of such apps in a regulation.
Whether health apps comply with the FADP must be assessed on a case-by-case basis. According to the Federal Data Protection and Information Commissioner, the decentralised privacy-preserving proximity tracing app (DP-3T, table 1) complies with the FADP .
Aim and purpose of the classification framework
The suggested framework aims to enable end users in critically assessing the purposes, as well as evaluating the trustworthiness, of COVID-19 related smartphone and web-based health apps. Based on these aims, the focus of our attempt lies less on usability and technical features, but more on transparency, scientific backing of purported aims, user privacy and data ownership. The framework is tailored to Swiss privacy law aspects and thus primarily geared to the Swiss legal setting. Nonetheless, it may also be equally applicable in other countries.
The present effort emerged out of discussions between Faculty members of the Digital Society Initiative (DSI) at the University of Zurich and was triggered by the need for advice on specific digital health technologies that are currently being advertised or discussed in the media. Two Faculty members with expertise in Epidemiology and Digital Health (VVW) and Law and Medicine (KNV) took the lead on this initiative and defined the overall aim and scope of this effort. Other DSI members provided systematic feedback.
Due to the urgency of the current situation, the approach to framework development was abbreviated, but nevertheless systematic. Based on a preliminary search, the checklist for mHealth apps by van Haasteren and colleagues, which focuses on the trustworthiness of mHealth apps (with a focus on physical activity apps), was identified as a suitable starting point for a framework development . These authors defined trustworthiness as “attributes that compel an individual to consider another individual or entity worthy of their trust” . A similar focus on trustworthiness was deemed appropriate for our study objective, since the current epidemic imposes a need for accelerated app release and development schedules, inevitably limiting available time for systematic testing and assessment by end users. In addition, the proposed framework should primarily rely on information that is usually publicly available from developers and advertisements.
The initial development process started with a critical review of the checklist of van Haasteren (which had an emphasis on fitness apps) and by reducing the number of items (e.g., those that were primarily pertaining to physical activity apps). The selection process was conducted by the two lead authors. In order to tailor the checklist to a well-suited framework to evaluate the trustworthiness of health apps in the context of COVID-19, we searched the literature for additional assessment checklists. Searches were conducted in PubMed and Google Scholar, using a set of pre-selected keywords. These included: “mobile health”, “mHealth”, “health app”, “evaluation”, “framework”, “assessment”, “checklist”, “rating”, “quality assessment” and “guidelines”. All keywords were identified through a preliminary screening of previous literature reviews and were carefully combined to increase the searches’ sensitivity. To ensure timeliness, we limited our searches to the last 10 years. The full search strategy and flow chart are provided in appendix 1.
After title and abstract screening, the search yielded 37 + 2 potentially suitable publications. These were full-text screened by one author (VN). The final selection yielded seven publications, which were reviewed in detail by two authors (VN, VVW) [38–44]. In particular, these additional checklists provided suggestions for additional domains and assessment criteria to be integrated into the initial checklist by van Haasteren . When selecting additional criteria, more weight was given to aspects covering the domains of user control and privacy, whereas additional criteria regarding usability were given less consideration. Likewise, criteria that were specific to commercial products, wearable devices, specific brands, or pertaining to marketing or integration with social media were not considered for our framework because they were outside this study’s scope.
Based on this review process and the seven additional checklists, the initial framework was complemented by the lead authors with additional items that were deemed relevant to user control, privacy and the current COVID-19 context. Further additions were made by the lead authors based on the assessment of the Swiss legal context, their subject knowledge and as part of first internal test evaluations of digital health tools (not reported here). This framework was then critically reviewed by other DSI members who were previously not involved in the development process.
A review of the 12 domains (each including between 1 and 8 questions, totalling 43 items) of the van Haasteren checklist  revealed some domains that were considered less relevant in this particular context of COVID-19: “understandability”, “brand familiarity”, “societal influences”, and “autonomy” (which was merged with empowerment and renamed as “user control/ self-determination”). In addition, the domain “purpose” was added to allow a critical assessment of the stated goals of the digital health tool in the context of the current COVID-19. Overall, of the initial 12 domains included in , eventually 6 were included in the final framework (table 3). An additional domain “purpose” was included in order to allow an assessment of the health technology’s goals in the context of the current COVID-19 epidemic. Understanding the COVID-19 related purpose of an app is key for acceptance, as well as effective and adequate use.
|Purpose||The purpose of the app, including the collection and analysis of data, should be clearly stated. Ideally, the purpose and methods are backed by scientific evidence|
|Usability||Apps should be intuitive and simple to use for intended target group. Ideally, the app has been tested in its specific target population.|
|Information accuracy||The information collected by the app should be valid and reliable. Data provided to end user should be current and accurate. Evidence to this effect should be presented.|
|Organisational attributes / reputation||End user should know who is behind an app (in terms of technological development, funding, or endorsement).|
|Transparency||Data collection should be fit for purpose and as little as needed. This includes data gathered by integrated device sensors.|
End user should be informed about potential risks of app usage and further sharing of data.
|Privacy||End user should be informed about privacy policies and data protection measures in a clear, transparent fashion.|
|User control / self-determination||End user should be informed whether and how they can access and control their data. This includes procedures in case the end user terminates participation.|
After having defined the assessment domains, the specific checklist items of van Haasteren were reviewed. We deleted items that are not easily applicable to mobile health apps or not answerable based on public data. In case of redundancies with other items, we tried to combine and re-formulate the items. The process of domain and item reduction is made transparent in supplementary material 2. It consisted of three steps, starting with initial additions and deletions in the original van Haasteren list (step 1), followed by an internal pre-test and the addition of further items (step 2), and concluded with a final item rearrangement and wording changes (step 3).
The final assessment framework is illustrated in table 4 (and provided as an excel table in appendix 2). It includes a total of 40 questions, which can be answered by “yes”, “no”, “unclear”, “in progress”, and “not applicable”. In addition to the specific items, the table further includes input fields for a short description of the purpose of the digital health tool, the postulated target population, and the sources (e.g., internet links) that were used for the assessment.
The list of questions is followed by a field for a qualitative statement regarding the use of the app in the context of the current COVID-19 pandemic, following the explanations provided in the introductory section of this manuscript. In particular, the assessment could facilitate the evaluation of an app’s potential to contribute to different epidemic management measures, such as those listed in table 2.
Yet, even during emergencies, it is essential to safeguard personal rights and privacy. Furthermore, clear and easy-to-understand information on the legal aspects of privacy, data collection and use are a good indicator of trustworthiness and will ultimately facilitate technology acceptance. Therefore, an evaluation of the legal embedding and the control mechanisms to protect a user’s identity and data ownership are of utmost importance, even more so if health-related information is collected. To this end, a dedicated section specifically assesses the legal embedding of the app. These items, demanding qualitative assessments, require substantial subject knowledge and may not be suitable or needed for all evaluations. Nevertheless, they should form an integral part of systematic COVID-19 app reviews. To this end, the framework outlines, for example, basic questions regarding the storage and use of collected data, data ownership and access rights, or the transfer of information to third parties.
Finally, the assessment framework concludes with an overall statement regarding the specific contribution of digital health technology in mitigating the current (or future) epidemics, having the potential target populations in mind. This latter aspect is very relevant because technologies that do not manage to catch on in the target group will not be able to create desired benefits. This becomes even more of an issue if the benefits are directed at special populations such as minors or the elderly (which may or may not overlap with the target population of end users as envisioned by the app developers).
|Table 4: Framework checklist.|
|Purpose (2-3 Sentences)|
|Included Information Sources|
|Purpose||1) Does the app have a clearly stated, understandable purpose for data collection and analysis?|
|2) Is the postulated purpose backed by scientific research?|
|3) Are the intended goals achievable, measureable?|
|4) Is regular feedback provided on the achievement of goals?|
|5) On which level are the stated benefits: |
a. personal level?
b. societal or population level?
|6) Do the stated benefits depend on a certain number of persons using the app?|
|Usability ||7) What is the evidence that the app is easy to use and has a friendly end-user interface?|
|8) Does the app send out a reasonable number of notifications?|
|9) Is the app accessible / understandable by its target audience?|
|10) Has the app been trialled / tested in its target audience?|
|Information accuracy ||11) Does the app create accurate measurements?|
|12) Does the app ensure that personalised data tailored to end-users are precise?|
|13) Is the information gathered and/or provided by the app backed by robust research?|
|14) Does the app provide regular updates to contents based on improved research and recommendations?|
|Organizational attributes / Reputation ||15) Does the company curating the app have clear policies on how to handle end-user data?|
|16) Is the app profit-oriented?|
|17) Is the app affiliated with a non-governmental organization or a reputable government agency?|
|18) Is the manufacturer of the app based in Switzerland?|
|19) Are all data stored and processed in Switzerland?|
|20) Has the manufacturer developed similar apps in the past?|
|Transparency ||21) Does the app inform the end user about the voluntary nature to participate?|
|22) Does the app highlight potential risks or side effects resulting from its use?|
|23) Does the app inform the end user what data is being collected?|
|24) Does the app require only minimal personal data of end-users?|
|25) Does the app require only minimal access permissions for using mobile phone functionalities (e.g. location, address book, camera)?|
|26) Is the source code publicly available?|
|27) Does the app integrate third-party software or use third-party APIs?|
|Privacy ||28) Are the privacy policies concise, clear and easy to understand?|
|29) Is the data generated from the app secured by end-to-end-encryption?|
|30) How is the data generated from the app stored:|
a. locally on the device?
|31) Is the data generated from the app anonymised for storage or analysis so that individuals are non-identifiable?|
|32) How is the data anonymized?|
|33) How long is the generated data stored?|
|User control/ Self-Determination||34) Do end-users act as the proprietors of the data generated from the app?|
|35) Does the app allow end-users to opt-in and decide which data can be stored or processed?|
|36) Does the app allow users to retrieve their data?|
|37) Can users get information about the results of the data analysis?|
|38) Does the app seek explicit end-user permission before sharing data with third-parties?|
|39) Does the app allow end-users to easily delete their data?|
|40) Do the users have the option to opt-out anytime and, if the case, is it clearly described what happens to the data?|
|Assessment of the subject specific basis of this app re. purpose, evidence base (2-3 sentences)|
|Assessment of legal basis of this app re. compliance with the Swiss regulations (2-3 sentences)|
|Overall assessment: For whom may this app be suited and what can it potentially contribute to mitigate the COVID-19 epidemic?|
Having applied a structured approach, we generated an assessment framework for smartphone and web-based health app evaluations in the context of the current COVID-19 pandemic. Starting with a checklist developed by van Haasteren et al. , we systematically reduced, combined and added items in order to create a set of questions that are generic but should nevertheless be answerable through publicly available information, as provided by app developers and manufacturers. Although developed for the Swiss setting and referencing Swiss laws, the framework is likely transferrable to other settings, particularly within Europe. Swiss data protection laws are similar to the General Data Protection Regulation (GDPR) that regulates the processing of data relating to individuals in the European Union .
Given the plans for widespread application of, for example, proximity tracing technologies, frameworks as ours are clearly relevant to engage the public in the discussion about benefits and risks of specific apps. The present framework complements existing tools by integrating an assessment of an app’s contribution to epidemic control mechanisms. Our framework shares similar domains with other checklists and frameworks that focus on security risks, privacy or manufacturer reputation, but is less focused on questions regarding app popularity (e.g., a user’s likeliness to recommend the app) or validity and measurement reliability, which are somewhat less relevant or unknown in the context of the current COVID-19 pandemic. In addition, our framework places less emphasis on usability and attractive design than other checklists. Although it is undeniable that these factors are crucial for a wide dissemination of apps and long-term user retention, we argue that, in the current state of emergency, users may be more likely to decide on an app’s use based on its intended purpose, likely rendering design aspects of secondary importance.
Our approach is subject to some limitations. Because of time pressure, the methods for framework development needed to deviate from standard development and literature review procedures. The literature search, although performed with great care, could not be as exhaustive as would normally be considered standard. Furthermore, the domain definition and item reductions were primarily performed by just two reviewers. This procedural limitation was somehow mitigated by the final review and revision process, which involved a larger group of experts. Because of these limitations, the framework should not be considered definitive, especially since only a small number of test evaluations could be performed in the short amount of available time. Nevertheless, we believe that the framework offers a valuable and highly informative starting point for initial assessments of emerging smartphone and web-based apps to control the current COVID-19 pandemic.
To summarise, the present framework synthesises several well-established app assessment domains, providing exhaustive guidance on evaluating an app’s trustworthiness, epidemiological rationale and legal robustness. Ideally, the specific questions of the framework should be answerable by end users, based on publicly available information, such as manufacturer websites and documentations. However, answering some of the framework’s items is likely to additionally require some expert knowledge. In combination with the framework’s current length, we believe that an abbreviated list, along with more extensive instructions on how to complete the items, will be required. The development of such a simplified end-user checklist is envisioned as a follow-up project.
Furthermore, the framework also provides some guidance for manufacturers on what information would be desirable in order to facilitate transparency and trustworthiness, both of which are key in times of crisis and uncertainty. Ultimately, the framework should aid individual decision making and steer the public’s attention towards crowdsourced digital health efforts, such as proximity tracing apps. Many of these efforts might have the potential to facilitate epidemic management while preserving privacy and individual user rights. Frameworks like ours aim to support the public in identifying such apps and keeping them apart from non-transparent and less trustworthy digital health solutions.
No financial support and no other potential conflict of interest relevant to this article was reported.
Prof. Viktor von Wyl, PhD, Digital and Mobile Health Group, Epidemiology, Biostatistics and Prevention Institute, Hirschengraben 84, CH-8001 Zurich, Viktor.vonwyl[at]uzh.ch
1 Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20(5):533–4. Published online February 23, 2020. doi:. http://dx.doi.org/10.1016/S1473-3099(20)30120-1 PubMed
2 Salathé M, Althaus CL, Neher R, Stringhini S, Hodcroft E, Fellay J, et al.COVID-19 epidemic in Switzerland: on the importance of testing, contact tracing and isolation. Swiss Med Wkly. 2020;150:w20225. doi:. http://dx.doi.org/10.4414/smw.2020.20225 PubMed
4 Sun K, Chen J, Viboud C. Early epidemiological analysis of the coronavirus disease 2019 outbreak based on crowdsourced data: a population-level observational study. Lancet Digit Health. 2020;2(4):e201–8. doi:. http://dx.doi.org/10.1016/S2589-7500(20)30026-1 PubMed
5 Wakefield J. Coronavirus: Tracking app aims for one million downloads. BBC; 2020 [cited 2020 April 16]. Available from: https://www.bbc.com/news/technology-52033210.
6 Buckee CO, Balsari S, Chan J, Crosas M, Dominici F, Gasser U, et al.Aggregated mobility data could help fight COVID-19. Science. 2020;368(6487):145–6. doi:. http://dx.doi.org/10.1126/science.abb8021 PubMed
7 Richardson DB, Kwan M-P, Alter G, McKendry JE. Replication of scientific research: addressing geoprivacy, confidentiality, and data sharing challenges in geospatial research. Ann GIS. 2015;21(2):101–10. doi:. http://dx.doi.org/10.1080/19475683.2015.1027792
8 Kounadi O, Leitner M. Why does geoprivacy matter? The scientific publication of confidential data presented on maps. J Empir Res Hum Res Ethics. 2014;9(4):34–45. doi:. http://dx.doi.org/10.1177/1556264614544103 PubMed
10 Wu Z, McGoogan JM. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72 314 Cases From the Chinese Center for Disease Control and Prevention. JAMA. 2020;323(13):1239. doi:. http://dx.doi.org/10.1001/jama.2020.2648 PubMed
11 Guan WJ, Ni ZY, Hu Y, Liang WH, Ou CQ, He JX, et al.; China Medical Treatment Expert Group for Covid-19. Clinical Characteristics of Coronavirus Disease 2019 in China. N Engl J Med. 2020;382(18):1708–20. doi:. http://dx.doi.org/10.1056/NEJMoa2002032 PubMed
12 Gengler I, Wang JC, Speth MM, Sedaghat AR. Sinonasal pathophysiology of SARS-CoV-2 and COVID-19: a systematic review of the current evidence. Laryngoscope Investig Otolaryngol. 2020:lio2.384. doi:. http://dx.doi.org/10.1002/lio2.384
13 Li Q, Guan X, Wu P, Wang X, Zhou L, Tong Y, et al.Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus-Infected Pneumonia. N Engl J Med. 2020;382(13):1199–207. doi:. http://dx.doi.org/10.1056/NEJMoa2001316 PubMed
14 Du Z, Xu X, Wu Y, Wang L, Cowling BJ, Meyers LA. Serial Interval of COVID-19 among Publicly Reported Confirmed Cases. Emerg Infect Dis. 2020;26(6). doi:. http://dx.doi.org/10.3201/eid2606.200357
16 He X, Lau EHY, Wu P, Deng X, Wang J, Hao X, et al.Temporal dynamics in viral shedding and transmissibility of COVID-19. Nat Med. 2020;26:672–5. doi:. http://dx.doi.org/10.1038/s41591-020-0869-5 PubMed
17 Rothe C, Schunk M, Sothmann P, Bretzel G, Froeschl G, Wallrauch C, et al.Transmission of 2019-nCoV Infection from an Asymptomatic Contact in Germany. N Engl J Med. 2020;382(10):970–1. doi:. http://dx.doi.org/10.1056/NEJMc2001468 PubMed
18 Kimball A, Hatfield KM, Arons M, James A, Taylor J, Spicer K, et al.; Public Health – Seattle & King County; CDC COVID-19 Investigation Team. Asymptomatic and Presymptomatic SARS-CoV-2 Infections in Residents of a Long-Term Care Skilled Nursing Facility - King County, Washington, March 2020. MMWR Morb Mortal Wkly Rep. 2020;69(13):377–81. Published online April 03, 2020. doi:. http://dx.doi.org/10.15585/mmwr.mm6913e1 PubMed
19 Wei WE, Li Z, Chiew CJ, Yong SE, Toh MP, Lee VJ. Presymptomatic Transmission of SARS-CoV-2 - Singapore, January 23-March 16, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(14):411–5. Published online April 10, 2020. doi:. http://dx.doi.org/10.15585/mmwr.mm6914e1 PubMed
20 Zou L, Ruan F, Huang M, Liang L, Huang H, Hong Z, et al.SARS-CoV-2 Viral Load in Upper Respiratory Specimens of Infected Patients. N Engl J Med. 2020;382(12):1177–9. doi:. http://dx.doi.org/10.1056/NEJMc2001737 PubMed
21 Zhang S, Diao M, Yu W, Pei L, Lin Z, Chen D. Estimation of the reproductive number of novel coronavirus (COVID-19) and the probable outbreak size on the Diamond Princess cruise ship: A data-driven analysis. Int J Infect Dis. 2020;93:201–4. doi:. http://dx.doi.org/10.1016/j.ijid.2020.02.033 PubMed
22 Ferretti L, Wymant C, Kendall M, Zhao L, Nurtay A, Abeler-Dörner L, et al.Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science. 2020;368(6491):eabb6936. doi:. http://dx.doi.org/10.1126/science.abb6936 PubMed
23 Mizumoto K, Kagaya K, Zarebski A, Chowell G.Estimating the Asymptomatic Proportion of 2019 Novel Coronavirus onboard the Princess Cruises Ship, 2020. medRxiv. 2020:20025866. doi: . http://dx.doi.org/10.1101/2020.02.20.20025866
24 Hellewell J, Abbott S, Gimma A, Bosse NI, Jarvis CI, Russell TW, et al.; Centre for the Mathematical Modelling of Infectious Diseases COVID-19 Working Group. Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. Lancet Glob Health. 2020;8(4):e488–96. doi:. http://dx.doi.org/10.1016/S2214-109X(20)30074-7 PubMed
25 Koppeschaar CE, Colizza V, Guerrisi C, Turbelin C, Duggan J, Edmunds WJ, et al.Influenzanet: Citizens Among 10 Countries Collaborating to Monitor Influenza in Europe. JMIR Public Health Surveill. 2017;3(3):e66. Published online September 21, 2017. doi:. http://dx.doi.org/10.2196/publichealth.7429 PubMed
26 Guerrisi C, Turbelin C, Blanchon T, Hanslik T, Bonmarin I, Levy-Bruhl D, et al.Participatory Syndromic Surveillance of Influenza in Europe. J Infect Dis. 2016;214(suppl_4):S386–92. doi:. http://dx.doi.org/10.1093/infdis/jiw280 PubMed
27 Milano B. How much access to data should be permitted during the COVID-19 pandemic? 2020 [16.04.2020]. Available from: https://today.law.harvard.edu/how-much-access-to-data-should-be-permitted-during-covid-19-pandemic/.
28 Federal Data Protection and Information Officer. A Few Facts about the Federal Act on Data Protection 2020 [10.05.2020]. Available from: https://www.edoeb.admin.ch/edoeb/en/home/the-fdpic/legal-framework/ii--a-few-facts-about-the-federal-act-on-data-protection.html.
29 Vokinger KN, Stekhoven DJ, Krauthammer M. Lost in Anonymization - A Data Anonymization Reference Classification Merging Legal and Technical Considerations. J Law Med Ethics. 2020;48(1):228–31. Published online April 29, 2020. doi:. http://dx.doi.org/10.1177/1073110520917025 PubMed
30 Federal Data Protection and Information Officer. Latest News 2020 [10 May 2020]. Available from: https://www.edoeb.admin.ch/edoeb/en/home/latest-news/aktuell_news.html#-629760265.
31 Swiss Confederation. Federal Act on Data Protection (FADP) 2019 [16 April 2020]. Available from: https://www.admin.ch/opc/en/classified-compilation/19920153/index.html.
32 Rudin B. Das Recht auf Anonymität, Anonymität als Teil der informationellen Selbstbestimmung: wenig geregelte Anwendungsfälle und viel Handlungsbedarf. Digma.2008;8:6–13.
33 Rosenthal D, Jöhri Y. Handkommentar zum Datenschutzgesetz sowie weiteren, ausgewählten Bestimmungen. Zurich: Schulthess Juristische Medien AG; 2008.
34 Maurer-Lambrou U, Gabor PB. Basler Kommentar, Datenschutzgesetz/Öffentlichkeitsgesetz. 3. Edition. Basel. 2014.
35 Vokinger KN. Gesundheitsdaten im digitalen Zeitalter 2020 [16.04.2020]. Available from: https://jusletter.weblaw.ch/fr/juslissues/2020/1008/gesundheitsdaten-im-_bf12b7abee.html__ONCE&login=false.
36 Vokinger KN. Digitale Bekämpfung von Covid-19 und die Rolle des Bundes(-rates). Eine rechtliche Würdigung des Einsatzes von Software. Submitted Manuscript. 2020.
37 Pärli K, Baeriswyl B. Contact-Tracing: Nur die staatliche App schafft Vertrauen. Neue Zürcher Zeitung [Internet]. 2020 16.04.2020. Available from: https://www.nzz.ch/meinung/contact-tracing-nur-die-staatliche-app-schafft-vertrauen-ld.1550670?reduced=true.
38 Albrecht UV, Malinka C, Long S, Raupach T, Hasenfuß G, von Jan U. Quality Principles of App Description Texts and Their Significance in Deciding to Use Health Apps as Assessed by Medical Students: Survey Study. JMIR Mhealth Uhealth. 2019;7(2):e13375. doi:. http://dx.doi.org/10.2196/13375 PubMed
41 Albrecht U-V, Pramann O, von Jan U. Medical Apps – The Road To Trust. European Journal for Biomedical Informatics.2015;11(3):7–12. doi:. http://dx.doi.org/10.24105/ejbi.2015.11.3.3
42 Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3(1):e27. doi:. http://dx.doi.org/10.2196/mhealth.3422 PubMed
43 Wyatt JC, Thimbleby H, Rastall P, Hoogewerf J, Wooldridge D, Williams J. What makes a good clinical app? Introducing the RCP Health Informatics Unit checklist. Clin Med (Lond). 2015;15(6):519–21. doi:. http://dx.doi.org/10.7861/clinmedicine.15-6-519 PubMed
44 Grundy QH, Wang Z, Bero LA. Challenges in Assessing Mobile Health App Quality: A Systematic Review of Prevalent and Innovative Methods. Am J Prev Med. 2016;51(6):1051–9. doi:. http://dx.doi.org/10.1016/j.amepre.2016.07.009 PubMed
45 European Commission. What does the General Data Protection Regulation (GDPR) govern? [10.05.2020]. Available from: https://ec.europa.eu/info/law/law-topic/data-protection/reform/what-does-general-data-protection-regulation-gdpr-govern_en.
Search ((((((((((mHealth[Title]) OR (mobile[Title] AND health[Title])) OR (smartphone[Title] AND health[Title])) OR (health[Title] AND app*[Title])) OR (disease[Title] AND app*[Title])))) AND (((((((((((((((Evaluat*[Title] AND list[Title])) OR (Evaluat*[Title] AND framework[Title])) OR (Evaluat*[Title] AND model[Title])) OR (Evaluat*[Title] AND systematic[Title])) OR (Evaluat*[Title] AND scale[Title])) OR (Classif*[Title] AND framework[Title])) OR (Assess*[Title] AND list[Title])) OR (Assess*[Title] AND framework[Title])) OR (Quality[Title] AND assess*[Title])) OR Rate[Title]) OR Rating[Title]) OR Checklist[Title]) OR Check-list[Title]) OR Guideline*[Title]))) AND "last 10 years"[PDat]) Filters: published in the last 10 years
Final assessment framework.
This appendix is available in a separate file at https://smw.ch/article/doi/smw.2020.20282.
Published under the copyright license CC BY-NC-SA: This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.