POLICY BRIEF
published: 21 April 2022
doi: 10.3389/fpubh.2022.832870
Frontiers in Public Health | www.frontiersin.org 1April 2022 | Volume 10 | Article 832870
Edited by:
Peter Kokol,
University of Maribor, Slovenia
Reviewed by:
Jernej Zavrsnik,
Community Health Centre, Slovenia
Florian P. Limbourg,
Hannover Medical School, Germany
*Correspondence:
Hendrikje Lantzsch
Specialty section:
This article was submitted to
Digital Public Health,
a section of the journal
Frontiers in Public Health
Received: 10 December 2021
Accepted: 16 March 2022
Published: 21 April 2022
Citation:
Lantzsch H, Panteli D, Martino F,
Stephani V, Seißler D, Püschel C,
Knöppler K and Busse R (2022)
Benefit Assessment and
Reimbursement of Digital Health
Applications: Concepts for Setting Up
a New System for Public Coverage.
Front. Public Health 10:832870.
doi: 10.3389/fpubh.2022.832870
Benefit Assessment and
Reimbursement of Digital Health
Applications: Concepts for Setting
Up a New System for Public
Coverage
Hendrikje Lantzsch1*, Dimitra Panteli2, Filippo Martino3, Victor Stephani4, David Seißler5,
Constanze Püschel6, Karsten Knöppler5and Reinhard Busse1
1Department of Health Care Management, Technische Universität Berlin, Berlin, Germany, 2European Observatory on Health
Systems and Policies Brussels, Brussels, Belgium, 3German Society of Digital Medicine e.V. (DGDM), Berlin, Germany,
4HelloBetter—GET.ON Institut für Online Gesundheitstrainings GmbH, Berlin, Germany, 5fbeta GmbH, Berlin, Germany,
6D+B Rechtsanwälte, Berlin, Germany
In Germany, some digital health applications (DiHA) became reimbursable through the
statutory health insurance system with the adoption of the Digital Healthcare Act in
2019. Approaches and concepts for the German care context were developed in an
iterative process, based on existing concepts from international experience. A DiHA
categorization was developed that could be used as a basis to enable the creation
of a reimbursed DiHA repository, and to derive evidence requirements for coverage
and reimbursement for each DiHA. The results provide an overview of a possible
classification of DiHA as well as approaches to assessment and evaluation. The structure
of remuneration and pricing in connection with the formation of groups is demonstrated.
Keywords: digital health applications, digital health technologies, medicine apps, benefit assessment,
reimbursement
INTRODUCTION
Digital health applications (DiHA), often also called mHealth applications, are cooperative and/or
interactive applications of modern information and communication technologies, aiming to
improve care and population health. They have rapidly proliferated in recent years, leading to
considerable transformation in the delivery of health services (1,2). DiHA have a wide range of
different functions and are used in various areas (3). The heterogeneous structures of health systems
and variability of regulations between countries mean that DiHA face different evaluation processes
before inclusion into the benefits basket.
The European Commission Expert Panel on effective ways of investing in health (4) highlighted
that the implementation, use, or reimbursement of DiHA should ideally be based on the evidence
about their performance: DiHA should be assessed against health system objectives such as access,
quality, and contribution to population health, efficiency, and equity. The proof of patient benefit
is also a central element in the DiHA evaluation framework of the World Health Organization to
conducting research and assessment (5). The benefit for the end user should be described and it
should be demonstrated why the application is innovative, and to what extent, it is superior to
Lantzsch et al. Assessment of Digital Health Applications
the standard of care. At the same time, approaches for
demonstrating the benefit of DiHA are increasingly emerging,
aiming to both do justice to the particular characteristics of these
health technologies and guarantee sufficient evidence to satisfy
the evidence standards described above (6).
European countries apply different regulations for the
payment of DiHA; they can partly be paid voluntarily by
patients and individual health insurees, or individual DiHA
with certain proof of patient benefit are paid obligatorily
by all health insurers. Examples of countries moving in
the latter direction with interesting developments in the
categorization of DiHA include Belgium (7), England (8,9),
and France (10,11). This is linked with an increased interest
in reimbursement options (12–14) or assessment frameworks
of DiHA (15,16). Germany however is considered a pioneer
in the European context, as a statutory reimbursement
obligation has already been introduced (17). To bring
DiHA into the statutory health insurance (SHI) system in
Germany, the Digital Health Care Act (Digitale-Versorgung-
Gesetz, DVG) was passed in November 2019 (18). The law
created, among other things, the entitlement of insured
persons to certain types of DiHA and broadly regulates the
reimbursement claims of DiHA, and the requirements that
manufacturers must fulfill to be eligible for funding. The
subsequent Digital Health Applications Ordinance (Digitale
Gesundheitsanwendungen-Verordnung, DiGAV) describes
the procedure and specific requirements for manufacturers in
detail (19).
Briefly put, to be reimbursed by SHI in Germany, a
DiHA must be classified as a class I or IIa medical device
according to Regulation (EU) 2017/745 on medical devices or
according to Directive 93/42/EEC, serve a specific purpose,
and not be used exclusively by healthcare providers/health
professionals. DiHA developers must submit an application
to the German Institute for Medicines and Medical Devices
(Bundesinstitut für Arzneimittel und Medizinprodukte, BfArM),
demonstrating that their application meets requirements for
safety, quality, functionality, privacy, and data security. They
must also demonstrate that the application has positive
care effects.
It has been argued that DiHA differ substantially from
both medicines and medical devices in their characteristics,
and this should be considered in the design of evidence-
based pathways toward reimbursement and pricing (20).
Perhaps most importantly, DiHA can reflect a care
process rather than a single product. At the same time,
DiHA compete for the same pot of funds as other health
interventions, and the principles underpinning decisions
on their reimbursement should be comparable. This article
addresses the question of how these supposedly contradictory
statements can be united. In the following, approaches
developed in the “I.DiGA project” (see Methods Section,
next) for a benefit assessment and pricing mechanism for
DiHA will be presented. While the project focused on the
German context, the approaches described in the following
may be useful for other countries considering setting up
similar systems.
METHODS
Parallel to the German “Digital Health Care” (DVG) legislative
process, the “I.DiGA project,” funded by the Federal Ministry of
Health, developed approaches for the categorization, evaluation,
and pricing/reimbursement of DiHA. The project took place in
parallel with the preparation, discussion, and adoption of the
DVG and related regulatory documents, and its results served
as one of the bases that informed this process. The project had
an agile design: contents were developed drawing on existing
international frameworks and initiatives, literature research, and
expert interviews. These contents were then discussed in seven
expert workshops involving a broad range of stakeholders to
identify areas in need of refinement and additional analysis, and
possible solutions were developed. There were three main areas
of investigation: (a) the development of a conceptual model for a
DiHA directory that could serve reimbursement purposes; (b) the
development of a framework for the evidence-based evaluation of
DiHA, encompassing process and methodological requirements;
and (c) the development of a remuneration system for DiHA,
including a pricing approach.
RESULTS
Developing a “Positive List” for DiHA:
Proposal for a DiHA Directory
Given the heterogeneity of DiHA, as well as the availability of
several DiHA serving the same purpose, it was necessary to
create a taxonomy that would facilitate reimbursement, pricing,
and prescribing decisions. We developed a categorization that
could serve as the basis for a directory with a triple functionality:
grouping DiHA in a meaningful way to enable the determination
of the type of evidence required for reimbursability; to
identify comparators, against which additional benefits can be
demonstrated; and to enable DiHA comparisons for health
professionals looking to choose which alternative to prescribe.
We proposed a “directory for digital health applications”
that groups DiHA according to the following attributes: (i)
application area, (ii) target group, (iii) function, and (iv) user. It
is listed in Table 1.
The application area (i) is derived from the 3-digit ICD 10
code of the disease for which the DiHA is to be used according to
its certification and this, together with the function, reflects the
intended purpose of the DiHA as a medical device. The target
group (ii) complements the scope of application and reflects
the disease severity or vulnerability of the patients who are to
receive the DiHA. The classification varies because the evidence
requirements for healthy people may be lower than for highly
vulnerable people. The function (iii) is divided into detection,
monitoring, and treatment (including alleviation), which are
further differentiated according to their exact functionality into
(4) Diagnosis, (5) Simple Monitoring, (6) Complex Monitoring,
(7) Indirect Intervention, and (8) Direct Intervention. Under
“function,” Categories 0–3 represent possible functionalities of
DiHA as medical devices according to Regulation (EU) 2017/745,
which are, however, not among those meeting the requirements
of the 2020 iteration of the DVG for prescribability (these were
Frontiers in Public Health | www.frontiersin.org 2April 2022 | Volume 10 | Article 832870
Lantzsch et al. Assessment of Digital Health Applications
TABLE 1 | Proposal for DiHA directory with criterion values and correspondence to evidence requirement levels (blue shading).
i. Scope (according to ICD-10) ii. Target group iii. Function of the DiHA iv. User Seq. No.
... 0Healthy without
known risk factors
0Documentation 1Only patients or relatives 001
002
003
…
Exx Endocrine, nutritional and
metabolic diseases
1Healthy with risk
factors
1Information 2Patients and Service Providers 001
002
003
…
Fxx Mental and behavioral
disorders
2Acutely ill, not
life-threatening
2Prevention, prediction, prognosis 3Service providers only …
Hxx Diseases of the eye and ear 3Chronically ill,
stable state of
health
3Examination of a physiological or pathological process or condition …
Ixx Diseases of the circulatory
system
4Highly
vulnerable/
unstable health
status
Detection (diagnosis) 4Diagnosis …
Oxx Pregnancy, Birth and
Postpartum
Supervision (monitoring) 5Simple monitoring …
6Complex monitoring
… Treatment (therapy) 7Indirect intervention (self-management) …
8Direct intervention (change in health conditions)
Light blue filling =Low requirement level; Medium blue filling =Medium requirement level; Dark blue filling =High requirement level.
included in the model to ensure its sustained usability should the
scope of the legal framework be expanded in future). The user
(iv) of a DiHA may not only determine the risk associated with
its use, but also influence the evaluation design and the potential
access and remuneration pathways. For the user, a distinction
is made as to whether the DiHA is used (1) by the patient or
a relative only, or (2) by the patient and healthcare provider.
Category (3) is reserved for DiHA that are used exclusively
by health professionals and are therefore not covered by the
DVG (illustrated here for the same reason as Categories 0–3
under “function”).
For each criterion, a DiHA included in the SHI reimbursement
list would be assigned the matching digits depending on
taxonomic position, and those would be sequentially merged
into a 6-digit number. Additional digits are added at the end to
provide a serial number for each DiHA (product) within a single
taxonomic position (Table 1).
Evidence Requirements for the
Reimburseability of DiHA
The DiHA classification system described above can serve inter
alia to link the identified DiHA groups to the level of evidence
required for the DiHA to demonstrate positive care effects and
qualify for reimbursement. For reasons of usability, the I.DiGA
project proposed a simplified distinction between three evidence
requirement levels: low, medium, or high. The requirement level
for each DiHA group in the matrix is determined by the two
attributes “target group” and “function.” The darker the shade
of the cell in Table 1, the more rigorous the study design of
the evidence required for reimbursability. The final level of
evidentiary requirements for each group of DiHA is determined
by the highest level assigned to one of the two categories (e.g., a
direct intervention for stable chronic patients would fall into the
“high” evidence category, determined by the high assignment for
DiHA with this function).
To further specify the acceptable evidence per requirement
level, considerations from existing frameworks were combined
(5,8,10). Three fundamental elements of study design were
chosen to provide the necessary foundation and retain simplicity.
Expectations on study design increase with the level but the
comparison against a control group was considered essential
for all levels. Only DiHA in the low requirement category can
potentially rely on study designs wherein participants form
their own comparison group (before/after designs) to qualify
for reimbursability. A medium requirement level means it is
sufficient to present results from (quasi-)experimental studies
with a control group. While the randomized allocation to the
intervention and control group is not a prerequisite, all relevant
confounding variables should be identified and considered to
the maximum possible extent. The analysis must follow the
“Intention to treat” (ITT) principle. When DiHA fall into a high
requirement category, a randomized controlled study with an
analysis following the ITT principle is required (Table 2).
Acceptable Outcomes for Demonstrating the Effects
of DiHA
Due to their nature, DiHA can have health effects and other
effects. Taking the overall aim of improving patient-relevant
Frontiers in Public Health | www.frontiersin.org 3April 2022 | Volume 10 | Article 832870
Lantzsch et al. Assessment of Digital Health Applications
outcomes, the project relied on the assumption that for the
overall effect of DiHA to be positive, either the health effects must
be better with the DiHA than without the DiHA, or the other
effects must be better if the health effects are the same (i.e., health
effects should not be jeopardized even if other effects are better;
the general interpretation of the DVG in Germany deviates from
this position, see Section Discussion).
For demonstrating the health effects of DiHA, end points that
reflect patient-relevant benefits in the form of an improvement
in health status, shortening of disease duration, prolongation
of life, reduction of side effects, or improvement in quality of
life are ideal. On the other hand, typical surrogate parameters
can also be acceptable under certain circumstances, as can
other intermediate end points such as medication adherence;
provided that their correlation with patient-relevant end points
is established (i.e., they have been validated).
Similar to the logic for determining the evidence requirement
levels described above, acceptable end points depend on the type
of DiHA as determined by its function and target group. Table 3
brings together these two attributes with the corresponding level
of required evidence and the respective acceptable end points for
determining health effects. For DiHA that make a diagnosis, take
over complex monitoring, or represent a direct intervention to
change health status, relevant health-related end points should
TABLE 2 | Study design characteristics per requirement level.
Low Middle High
Control group X (before-
after)
X X
Intention-to-treat
evaluation
X X
Randomized group
allocation
X
Light blue filling =Low requirement level;
Medium blue filling =Medium requirement level;
Dark blue filling =High requirement level.
be investigated with robust study designs. Surrogate parameters
can be used as end points for these DiHA only if non-vulnerable
target groups are involved. For the DiHA that correspond to
indirect interventions or simple monitoring, the spectrum of
health-related end points can be expanded to include other
intermediate outcomes.
As DiHA often reorganize whole processes of care, non-
health outcomes can be used to determine reimbursement,
provided that the health effects are not compromised. The field
of health technology assessment (HTA) has long considered the
potential implications of the application of health technologies
across a range of domains, and these can also be used to
derive outcome categories here (21). Specifically, organizational,
social/ethical, and economic aspects were considered particularly
relevant for determining a DiHA’s eligibility for reimbursement.
Building on the EUnetHTA Core Model R
, core aspects with
associated contextual questions can help identify possible
outcome areas and target variables (see Appendixes 1–5 in
Supplementary Material). As with health effects, the evaluation
of other care effects of DiHA should be based on a comparative
analysis, if they are to improve patient outcomes. While
established outcomes for the measurement of health and
economic effects are available, this is largely not the case
for organizational and social/ethical effects. How these can be
measured depends on function and target group and cannot be
determined uniformly.
Relationship Between Health Effects and Other Care
Effects
As described earlier, the I.DiGA project worked on the
assumption that other care effects should be used to determine
reimbursability only if sufficient evidence has shown that health
effects are superior, or at least not inferior to those of the
appropriate comparator. Drawing on similar considerations in
France (10), the matrix in Table 4 was designed to visualize the
interplay between health effects, other care effects, and overall
reimbursability. If neither the health effects nor the other care
TABLE 3 | Matrix for determining the requirement level and acceptable end points for health effects.
Categorization regarding
“target group”
Categorization regarding “function”
4 Diagnosis
8 Direct intervention
6 Complex monitoring 7 Indirect intervention 5 Simple monitoring
4 Highly vulnerable and/or
unstable health status
High High High
2 acutely ill, not life-threatening/
3 chronically stable
High Medium Medium
1 healthy with risk factors High Medium Low
Acceptable endpoints for health
care effects
Patient-relevant benefit Efficacy (surrogate parameters if applicable) other intermediate outcomes, e.g. adherence
Light blue filling =Low requirement level; Medium blue filling =Medium requirement level; Dark blue filling =High requirement level.
Frontiers in Public Health | www.frontiersin.org 4April 2022 | Volume 10 | Article 832870
Lantzsch et al. Assessment of Digital Health Applications
TABLE 4 | Relationship between health effects and other care effects.
Compared to an appropriate comparator Health effects regarding the
DiHA’s function/purpose
Other care effects (organizational,
social/ethical, economic)
Overall evaluation (eligibility
for reimbursement)
Superior + + +
+0+
+–+
Comparable 0+ +
0 0 –
0 – –
Worse – Not relevant –
TABLE 5 | Possible price tiers based on the criteria of study results, quality of evidence, and comparator.
Study design Study quality Type of positive effect Extent of the (additional) effect Comparator Price tier
Not appropriate* Irrelevant Not reimb.
Appropriate* Not according to
best practice
Health effect OR other care effect
only
Irrelevant Not reimb.
Health effect AND other care effect Irrelevant Tier 1
According to best practice Other care effect (health
comparable)
Irrelevant Tier 1
Health effect (and possibly other
care effect as well)
Small No DIHA or DiHA
tier 1
Tier 1
DiHA tier 2 Tier 2
Considerable/substantial No DiHA Tier 2
DiHA tier 1/2 Tier 3
DiHA tier 3 Tier 4
*Low/middle/high depending on the requirement according to Table 2.
effects are superior to a corresponding comparison, eligibility for
reimbursement should not be considered given.
Pricing and Remuneration Systems of
DiHA
Principle of Price Determination for DiHA
Based on the usual pricing methods applied or discussed for
health technologies, the amount public payers are to spend
on DiHA qualifying for reimbursement can be based on (a)
the extent of the positive effect proved in studies and/or
(b) development and production costs and/or (c) a price
comparison (national/European consumer prices or European
reimbursement prices). Depending on health system goals and
payer criteria, price negotiations can consider only positive health
effects, or also other care effects such as increased efficiency.
Based on any one or a combination of these three components,
an initial base amount to be paid for a DiHA can be calculated,
potentially complemented by performance-based remuneration
components for actual positive effects achieved by patients
using the DiHA. DiHA utilization data or patient-reported
outcome and/or experience measures can be used to determine
such components.
Considerations on a Remuneration Model
The proposed logic for a remuneration and pricing model builds
on the previous ideas of clustering DiHA based on indication,
function, and target group and requiring comparative evidence
before they can qualify for reimbursement. For DiHA with
comparable measured effects and underlying study quality, prices
should also be comparable. A new DiHA would only be eligible
for a higher price if it were superior in term of positive effects
and/or was supported by methodologically better studies than
already reimbursed DiHA in the same group. An additional
dimension that may contribute to a higher price within a DiHA
group concerns the comparator in the submitted studies: a
new DiHA that can show benefit compared to existing DiHA
would receive a higher price than DiHA that have shown benefit
compared only with no DiHA use. If the three criteria positive
effects (by type and extent), quality of evidence (including overall
study design and quality of individual studies), and comparator
are combined, a classification leading to different price tiers can
be created (Table 5).
DISCUSSION AND POLICY IMPLICATIONS
The aim of the I.DiGA project was to provide a basis for
a workable system of implementing DiHA in the public
reimbursement system, particularly for the German context.
With the DVG, Germany is considered a pioneer, where patient-
facing DiHA are now reimbursable by the SHI system. However,
the project aimed at providing conceptual guidance usable
Frontiers in Public Health | www.frontiersin.org 5April 2022 | Volume 10 | Article 832870
Lantzsch et al. Assessment of Digital Health Applications
beyond the context of the DVG as passed in 2020, regarding, for
instance, an expansion in the scope of regulated DiHA. It should
be helpful for relevant stakeholders thinking about how to set up
a public system for DiHA access in any country.
Several countries have developed or are developing
approaches to better understand and evaluate DiHA for
health system use; however, these are at different stages of
development and clearly dependent on the country’s health
system set-up and coverage decision-making processes so far
(4,22,23). Concerns regarding privacy, efficacy, usability, and
implementation of DiHA are common characteristics of all
these initiatives. These could potentially be alleviated by clear
evaluation guidelines from the relevant decision-makers, as they
have now been developed in Germany (24).
The system to determine the reimbursability of DiHA
presented in this paper has similarities with other approaches,
such as the Belgian validation pyramid (7), but also important
differences, such as the categorization based on target group
and function. The categorization matrix is more granular than
existing alternatives for a formal app repository, such as the
National Health Service (NHS) App (9). Regarding evidentiary
requirements and related pricing options, our model draws on
the NICE framework from England (8), and the Medical Device
and Health Technology Evaluation Committee (CNEDiMTS)
framework (10), where, however, the number of evidence levels
and the level of detail of reimbursement situations differ.
A systematic review of evaluation frameworks for
mobile health applications found that the majority of the
identified frameworks that examined safety—only a few
assessed potential harm and only one explicitly considered a
comparison group (25). However, it has been recognized that
health systems must provide adequate approaches for DiHA
evaluation and reimbursement (5,26), and HTA can provide a
methodological basis (27). To make informed decisions about
the reimbursement, prescription and use of DiHA, stakeholders,
clinicians, and users need guidance that adapts to diverse local
requirements, standards, and current content (28). To create a
standard set of requirements, state regulatory agencies should
collaborate with clinical stakeholders (22). Assessment tools for
DiHA that aim to measure risks and benefits are proliferating
(29). Methodological considerations for evaluating DiHA are
increasingly developed, for instance, primarily for specific
DiHA groups [e.g., for mental health, see (30), or with a more
comprehensive scope (31)].
The work in the I.DiGA project confirmed the national
and international relevance of the topic. Even though there
are more and more approaches toward DiHA evaluation and
reimbursement decisions, none have really been tested yet in
the real world. Consequently, the implementation of DiHA
reimbursement processes should be closely monitored and
evaluated, and shared in international dialogue. The concepts
developed in the I.DiGA project can provide an initial basis
but will likely need to be adapted once the implementation
of current provisions in Germany and other countries has
been evaluated. The exchange of knowledge and experience
at European level could contribute to the improvement and
expansion of relevant assessment frameworks; existing initiatives
should be supported, and funding should be made available for
implementation research.
The study was developed based on the regulations and
particularities of the German health system; this may be
considered a limitation, or at least a challenge for transferability.
However, even if other countries have different starting points,
our considerations can be adopted to some extent. Another
limitation of our work pertains to the selective consideration of
existing literature and country examples for the evaluation of
DiHA, as no systematic search was carried out. A number of
new publications may have become available since this work was
concluded. The insights in this paper are mainly conceptual in
nature and have not been validated yet; however, they partially
draw on existing and tested regulatory levers for other health
technologies in the German health system.
CONCLUSION
The I.DiGA project developed concepts to categorize, evaluate,
and price DiHA for reimbursement. These can be used
individually or in combination to inform the formalization
of any DiHA reimbursement system. Fundamentally, DiHA
that want to be covered by statutory health systems need
to provide robust evidence of patient benefit, which can be
used to determine reimbursement eligibility and price. As
other European countries are also increasingly developing
approaches for categorizing or reimbursing DiHA, cross-border
cooperation for the development of efficient evaluation methods
which consider the particularities of DiHA is indicated, and
has the potential of expanding reimbursement opportunities
for developers.
AUTHOR CONTRIBUTIONS
HL, DP, and RB wrote the manuscript. All authors conceived the
contents of the project and revised the manuscript. All authors
contributed to the article and approved the submitted version.
FUNDING
The I.DiGA project was funded by the German Federal Ministry
of Health (Grant Reference: ZMVI1-2519FSB800).
ACKNOWLEDGMENTS
We would like to thank Sabine Fuchs for her input, and
all participants who attended the expert workshops and gave
content-related impulses.
SUPPLEMENTARY MATERIAL
The Supplementary Material for this article can be found
online at: https://www.frontiersin.org/articles/10.3389/fpubh.
2022.832870/full#supplementary-material
Frontiers in Public Health | www.frontiersin.org 6April 2022 | Volume 10 | Article 832870
Lantzsch et al. Assessment of Digital Health Applications
REFERENCES
1. Thiel R, Deimel L, Schmidtmann D, Piesche K, Hüsing T, Rennoch J,
et al. Health System Comparison Focus Digitalization. #SmartHealthSystems.
International comparison of digital strategies. Bertelsmann Foundation
2018 (2019).
2. Williams GA, Fahy N. Building and maintaining public trust to support
the secondary use of personal health data. Eur Observ Health Syst Policies.
(2019) 25:7–10.
3. Fahy N, Williams GA. Use of digital health tools in Europe: before, during
and after COVID-19. Policy Brief. (2021) 42:34. Available online at: https://
eurohealthobservatory.who.int/publications/i/use-of-digital-health-tools-
in-europe-before-during-and-after-covid-19 (accessed December 9, 2021).
4. European Commission. Assessing the Impact of Digital Transformation of
Health Services. Report of the Expert Panel on effective ways of investing
in Health (EXPH). (2019). Available online at: https://ec.europa.eu/health/
expert_panel/sites/expertpanel/files/docsdir/022_digitaltransformation_en.
pdf (accessed December 9, 2021).
5. World Health Organization. Monitoring and Evaluating Digital
Health Interventions: A Practical Guide to Conducting Research and
Assessment (2016). Available online at: https://apps.who.int/iris/
bitstream/handle/10665/252183/9789241511766-eng.pdf;jsessionid=
BFA562487E3057944410F7A9A13219FD?sequence=1 (accessed December 9,
2021).
6. Dick S, O’Connor Y, Heavin C. Approaches to mobile health
evaluation: a comparative study. Inform Syst Manag. (2020) 37:75–92.
doi: 10.1080/10580530.2020.1696550
7. Mhealthbelgium. Validation Pyramid of mHealth, Belgium. (2022). Available
online at: https://mhealthbelgium.be/validation-pyramid (accessed February
4, 2022).
8. National Institute for Health and Care Excellence (NICE). Evidence Standards
Framework for Digital Health Technologies (2019). Available online at: https://
www.nice.org.uk/Media/Default/About/what-we-do/our-programmes/
evidence-standards-framework/digital-evidence-standards-framework.pdf
(accessed December 9, 2021).
9. NHS. NHS App. (2022). Available online at: https://www.nhs.uk/nhs-app/
(accessed February 4, 2022).
10. French National Authority for Health (HAS). Évaluation des dispositifs
médicauxpar la CNEDiMTSGuide sur les spécificités d’évaluation
clinique d’un dispositif médical connecté (DMC) en vue de sonaccès au
remboursement. (2019). Available online at: https://www.has-sante.fr/upload/
docs/application/pdf/2019-02/guide_sur_les_specificites_devaluation_
clinique_dun_dmc_en_vue_de_son_acces_au_remboursement.pdf (accessed
December 9, 2021).
11. L’Assurance Maladie. La liste des produits et prestations—LPP (2021). Available
online at: https://www.ameli.fr/medecin/exercice-liberal/remuneration/
nomenclatures-codage/liste-produits-prestations-lpp (accessed February 4,
2022).
12. Jeindl R, Wild C. Framework zur Unterstützung von
Refundierungsentscheidungen zu digitalen Gesundheitsanwendungen
(mHealth) und dessen (retrospektive) Anwendung an ausgewählten Beispielen.
AIHTA Projektbericht Nr.: 134; 2020. Wien: HTA Austria – Austrian Institute
for Health Technology Assessment GmbH (2020). Available online at: https://
eprints.aihta.at/1279/1/HTA-Projektbericht_Nr.134.pdf (accessed February
4, 2022).
13. Degelsegger-Márquez A, Panteli D. Towards reimbursement of
mobile digital health applications? An international comparison of
practices. Eur J Public Health. (2020) 30(Supplement_5):ckaa165-1327.
doi: 10.1093/eurpub/ckaa165.1327
14. European Med Tech and IVD Reimbursement Consulting. Reimbursement
Landscape for Health Apps in Europe. (2020). European Med Tech and IVD
Reimbursement Consulting.
15. Lagan S, Sandler L, Torous J. Evaluating evaluation frameworks: a scoping
review of frameworks for assessing health apps. BMJ Open. (2021) 11:e047001.
doi: 10.1136/bmjopen-2020-047001
16. Hensher M, Cooper P, Dona SWA, Angeles MR, Nguyen D, Heynsbergh
N, et al. Scoping review: Development and assessment of evaluation
frameworks of mobile health apps for recommendations to consumers.
J Am Med Inform Assoc. (2021) 28:1318–29. doi: 10.1093/jamia/oc
ab041
17. Gerke S, Stern AD, Minssen T. Germany’s digital health reforms
in the COVID-19 era: lessons and opportunities for other
countries. NPJ Digit Med. (2020) 3:94. doi: 10.1038/s41746-020-0
306-7
18. Gesetz für eine bessere Versorgung durch Digitalisierung und Innovation
(Digitale-Versorgung-Gesetz; DVG) (2019). SGB V; Bundesgesetzblatt
(2019), I, Nr. 49, S. 2562–2584 (2019). Available online at: https://www.bgbl.
de/xaver/bgbl/start.xav?startbk=Bundesanzeiger_BGBl&start=%2F%2F%2A
%5B%40attr_id=%27bgbl119s2562.pdf%27%5D#__bgbl__%2F%2F*%5B
%40attr_id%3D%27bgbl119s2562.pdf%27%5D__1610879899265 (accessed
December 9, 2021).
19. Bundesanzeiger Verlag. Verordnung über das Verfahren und die
Anforderungen zur Prüfung der Erstattungsfähigkeit digitaler
Gesundheitsanwendungen in der gesetzlichen Krankenversicherung (Digitale
Gesundheitsanwendungen-Verordnung - DiGAV) (2021). Available online at:
https://www.bgbl.de/xaver/bgbl/start.xav?startbk=Bundesanzeiger_BGBl&
amp;jumpTo$=$bgbl120s0768.pdf#__bgbl__%2F%2F*%5B%40attr_id%3D
%27bgbl120s0768.pdf%27%5D__1625152759885 (accessed December 9,
2021).
20. European Commission. Proposed Guiding Principles for Reimbursement
of Digital Health Products and Solutions. Prepared for adoption by a
SubGroup of the eHealth Stakeholder Group (2019). Available online
at: https://www.medtecheurope.org/wp-content/uploads/2019/04/30042019_
eHSGSubGroupReimbursement.pdf (accessed December 9, 2021).
21. EUnetHTA. EUnetHTA Joint Action 2, Work Package 8. HTA Core Model
R
version 3.0 (2016). Available online at: https://eunethta.eu/wp-content/
uploads/2018/03/HTACoreModel3.0-1.pdf (accessed December 9, 2021).
22. Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, Cohen
AB. Digital health: a path to validation. NPJ Digit Med. (2019) 2:38.
doi: 10.1038/s41746-019-0111-3
23. Karpathakis K, Libow G, Potts HWW, Dixon S, Greaves F, Murray E.
An evaluation service for digital public health interventions: user-centered
design approach. J Med Internet Res. (2021) 23:e28356. doi: 10.2196/
28356
24. Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM) Das
Fast Track Verfahren für digitale Gesundheitsanwendungen (DiHA)
nach § 139e SGB V. Ein Leitfaden für Hersteller, Leistungserbringer und
Anwender. (2020). Available online at: https://www.bfarm.de/SharedDocs/
Downloads/DE/Service/Beratungsverfahren/DiHA-Leitfaden.pdf?__blob=
publicationFile (accessed December 9, 2021).
25. Moshi MR, Parsons J, Tooher R, Merlin T. Evaluation of mobile
health applications: is regulatory policy up to the challenge? Int J
Technol Assess Health Care. (2019) 35:351–60. doi: 10.1017/S02664623190
00461
26. Alami H, Gagnon M-P, Fortin J-P. Digital health and the
challenge of health systems transformation. Mhealth. (2017) 3:31.
doi: 10.21037/mhealth.2017.07.02
27. Vukovic V, Favaretti C, Ricciardi W, Waure C de. Health technology
assessment evidence on e-health/m-health technologies: evaluating the
transparency and thoroughness. Int J Technol Assess Health Care. (2018)
34:87–96. doi: 10.1017/S0266462317004512
28. Camacho E, Hoffman L, Lagan S, Rodriguez-Villa E, Rauseo-Ricupero N,
Wisniewski H et al. Technology evaluation and assessment criteria for health
apps (TEACH-Apps): pilot study. J Med Internet Res. (2020) 22:e18346.
doi: 10.2196/18346
29. Levine DM, Co Z, Newmark LP, Groisser AR, Holmgren AJ, Haas JS et al.
Design and testing of a mobile health application rating tool. NPJ Digit Med.
(2020) 3:74. doi: 10.1038/s41746-020-0268-9
30. Torous J, Andersson G, Bertagnoli A, Christensen H, Cuijpers P, Firth J
et al. Towards a consensus around standards for smartphone apps and
digital mental health. World Psychiatry. (2019) 18:97–8. doi: 10.1002/wps.
20592
31. Henson P, David G, Albright K, Torous J. Deriving a
practical framework for the evaluation of health apps. Lancet
Digital Health. (2019) 1:e52–4. doi: 10.1016/S2589-7500(19)3
0013-5
Frontiers in Public Health | www.frontiersin.org 7April 2022 | Volume 10 | Article 832870
Lantzsch et al. Assessment of Digital Health Applications
Conflict of Interest: VS was employed by HelloBetter—GET.ON Institut für
Online Gesundheitstrainings GmbH. DS and KK were employed by fbeta GmbH.
CP was employed by D+B Rechtsanwälte.
The remaining authors declare that the research was conducted in the absence of
any commercial or financial relationships that could be construed as a potential
conflict of interest.
Publisher’s Note: All claims expressed in this article are solely those of the authors
and do not necessarily represent those of their affiliated organizations, or those of
the publisher, the editors and the reviewers. Any product that may be evaluated in
this article, or claim that may be made by its manufacturer, is not guaranteed or
endorsed by the publisher.
Copyright © 2022 Lantzsch, Panteli, Martino, Stephani, Seißler, Püschel,
Knöppler and Busse. This is an open-access article distributed under the terms
of the Creative Commons Attribution License (CC BY). The use, distribution
or reproduction in other forums is permitted, provided the original author(s)
and the copyright owner(s) are credited and that the original publication in
this journal is cited, in accordance with accepted academic practice. No use,
distribution or reproduction is permitted which does not comply with these
terms.
Frontiers in Public Health | www.frontiersin.org 8April 2022 | Volume 10 | Article 832870
