Developing a Theoretical Framework for Evaluating the Quality of mHealth Apps for Adolescent Users: A Systematic Review

Introduction

In the United States, over 80% of the nation's annual healthcare expenditure is spent on managing chronic health conditions.1 This presents an ongoing public health issue for both adults and adolescents. Indeed, 1 in 5 US adolescents between the ages of 12 and 17 years lives with a chronic health condition such as asthma, diabetes, or depression,2 and 1 in 3 lives with 3 or more chronic conditions.2

Self-management of chronic conditions often entails continuous activities such as consistent medication taking, symptom monitoring, and other condition-specific treatment activities. These tasks may be more challenging for adolescents compared with adults. Adolescents may be especially prone to barriers resulting from lack of social support, understanding about a prescribed treatment regimen, disease state knowledge, and self-efficacy for self-management behaviors.3–5 Further, adolescents living with chronic conditions consider friendships, support from family and healthcare professionals, and positive school experiences as key factors in successfully managing their health conditions.6 To promote effective and sustainable self-management behaviors in adolescents, there is a need for targeted solutions that consider the unique needs and challenges faced by adolescents living with chronic conditions.

One viable option for improving self-management across a range of chronic conditions in adolescents is to use mobile health (mHealth) application (app) technology for smartphones and tablets. Use of mobile apps may improve adolescent engagement with health behaviors.7 Indeed, interventions leveraging mobile apps or text-messaging are effective in improving a broad range of health behaviors in adolescents, including diabetes self-management,8 asthma self-management,9 mental health,10 and condition-specific medication adherence.11 Given that 73% of all US teenagers between 13 and 17 years of age own or have access to a smartphone and the familiarity of this age group with using mobile technology,12 public health programs and interventions disseminated via mobile apps may have increased “reach” within the adolescent population compared with paper-based or face-to-face programs.

Despite the benefits of using mobile apps for engaging adolescents in self-management of health behaviors, there is no published tool to guide the development and evaluation of the quality of mHealth apps designed specifically for adolescent users. Evaluation tools exist for rating the quality of mHealth apps, such as the Mobile App Rating Scale (MARS),13 but are not designed specifically to rate apps intended for the unique needs of adolescent end users. Developing a rating tool designed specifically for evaluation of mHealth apps for adolescent users is a critical first step toward widespread design and dissemination of high-quality, evidence-based apps for improving self-management behaviors, and ultimately patient outcomes in this vulnerable population.

The need for such a tool is high. However, little is known about adolescent users' preferences for features in mobile apps. Additionally, relevant evaluation criteria for inclusion in a rating tool that meets adolescents' preferences have not been defined. Thus, the purpose of this systematic review is to assess adolescents' preferred mobile app features and to propose a framework for developing health-related mobile apps for adolescents. A grounded theory approach was used to evaluate the reviewed articles and create a theoretical framework of rating criteria that can be used to inform development of a quality-rating tool for development, evaluation, and comparison of mHealth apps targeted to adolescent users. Developing an adolescent mHealth app to improve medication adherence for chronic conditions is presented as an illustrative example of how the proposed framework may be used.

Materials and Methods

Search Strategy. PubMed, CINAHL, PsycINFO, ERIC, HealthIT.gov, and ClinicalTrials.gov were systematically searched in August 2017 for articles pertaining to mHealth app design and evaluation in an adolescent population. Citations in pertinent articles were also hand-searched to identify relevant articles that may not be indexed in the included databases. Search terms included (“mobile app” OR app OR mobile) AND (criteria OR scale OR rating OR checklist OR design OR develop) AND (child OR adolescent OR teen) AND (health OR medical OR drug OR mHealth).

Study Selection. Articles underwent initial title and abstract screening, followed by full-text screening. Each of these screenings was performed independently by 2 of the authors (RJ and NH), with discrepancies in retained articles resolved through discussion and consensus. English-language articles were included for review if they met the following criteria: 1) published between 2007 and 2017; 2) peer-reviewed cross-sectional analytic studies, quantitative descriptive studies, qualitative interviews or focus groups, or other app development, feasibility, or usability assessments (impact studies, study protocols, ecologic momentary assessments, and review articles were excluded); 3) intended users of apps are between the ages of 12 and 18 years (adolescents) and are normally developing (studies involving adolescents with neurodevelopmental disorders such as autism were excluded); 4) inclusion of preferred app features and rating criteria for the app; 5) inclusion of smartphone or tablet mobile app running on any platform (web-based, text, and personal digital assistant (PDA) applications were excluded); and 6) ability to extract results (rating and design criteria).

Data Extraction and Synthesis. Data were extracted by RJ and NH during full-text screening, using a standardized form. This standardized form contains fields for study location, general description (including study design, sample size, and participant characteristics), objective, measures, and results, as well as app features and rating criteria reported within each study.

Quality assessment was independently performed by RJ and NH using the Mixed Methods Appraisal Tool Version 2011,14 simultaneous with data extraction. The Mixed Methods Appraisal Tool contains 5 study design categories: qualitative, quantitative randomized controlled, quantitative non-randomized, quantitative descriptive, and mixed-methods. Each category contains 4 criteria (or 3 for mixed-methods), and scores range from 0% to 100%. For qualitative, quantitative randomized controlled, quantitative non-randomized, and quantitative descriptive studies, scores are calculated as a percentage of criteria met. For mixed-methods studies, scores are calculated as the lowest score from among the 3 relevant designs (quantitative, qualitative, and mixed methods). Interrater reliability of quality assessment scores was analyzed in the form of percent agreement.

Qualitative synthesis of the data was performed independently by RJ and NH to develop initial themes reflecting adolescents' preferred mHealth app features and best practices for evaluating the quality of health-related mobile apps for adolescents. Initial themes were discussed and modified by RJ and NH until consensus was reached, followed by review and verification by a third author (BF), to generate final themes. With a grounded theory approach,15 these final themes were used to construct a theoretical framework of rating criteria that can be used to inform development of a quality rating tool for design, evaluation, and comparison of mHealth apps targeted to adolescent users.

Results

PRISMA. A total of 13 articles met the inclusion/exclusion criteria for this review. Study selection is depicted in Figure 1. The most common reasons for exclusion were not including an mHealth app (e.g., use of a text-based tool); not explicitly listing rating criteria; not focusing on development, design, or rating of apps; and non-eligible study design.

View Figure

• Download as Powerpoint
• Download Figure

Characteristics and Quality of Included Studies. Table 1 contains a list of included studies and a summary of their design and general characteristics. All studies included qualitative design elements; 7 of the 13 studies used a purely qualitative design,16–22 whereas the remaining 6 studies used a mixed-methods design including both qualitative and quantitative analyses.8,23–27 The main objective of all retained studies was to design, develop, and/or evaluate mHealth apps for adolescent use. Feasibility, usability, and/or functionality testing were common purposes of app evaluation. Eight studies focused on self-management of asthma,19–21 cancer,23,26,27 or type 1 diabetes mellitus8,16 in adolescents. The remaining 5 studies focused on adolescents' self-management of Sickle Cell Disease,24 sexually transmitted infections,22 lupus,18 mental health conditions,17 and medication adherence in solid organ transplant.25 All 13 studies directly included adolescents as participants (age, 12–18 years).8,16–27 Four studies also supplemented adolescent or patient results with parent or non-professional caregiver results.8,16,23,25

Table 1. Characteristics of Included Studies

View Fullscreen

Table 2 contains the quality assessment results. Quality assessment ratings ranged from 50% to 100%, with 7 studies achieving a rating of 75% to 100%.16–22 Interrater reliability of quality ratings was high (percent agreement, 82.86%).

Table 2. Article Quality Assessment

View Fullscreen

Table 2. Article Quality Assessment (cont.)

View Fullscreen

Table 2. Article Quality Assessment (cont.)

View Fullscreen

Table 2. Article Quality Assessment (cont.)

View Fullscreen

Qualitative Synthesis. App Features Preferred by Adolescents. The qualitative synthesis performed by RJ and NH revealed mHealth app features preferred by adolescents. Preferred features commonly mentioned include ability to track test results or self-management progress; customize the app; connect to social media; gain points or prizes through app gamification; and easily navigate and read the app (including visual appeal and emphasis on using graphics instead of text). Table 3 contains a complete list of preferences.

Table 3. Features of Mobile Apps Designed for Adolescent Use, and Rating Criteria for App Evaluation

View Fullscreen

Table 3. Features of Mobile Apps Designed for Adolescent Use, and Rating Criteria for App Evaluation (cont.)

View Fullscreen

App Rating Criteria. Many studies performed hands-on usability tests with adolescents to assess app quality in terms of meeting end users' needs for app functionality and likability. Methods of measuring app quality varied across studies, including qualitative discussion, survey measures using Likert-type scales, and/or other survey rating measures. Quality criteria most often mentioned included degree of app customizability, ease of use, visual appeal, interactivity (with peers, clinicians, or social media), self-management capability, and condition-specific features (Table 4).

Table 4. Dimensions for Rating the Quality of mHealth Apps Targeted to Adolescents

View Fullscreen

Theoretical Framework. Figure 2 presents a framework for developing health-related mobile apps for adolescents, based on qualitative synthesis of adolescent-preferred app features and common quality rating criteria from the included studies. Five dimensions emerged: 1) Technical Quality, including 8 constructs, such as app ease of use; 2) Engagement, including 6 constructs, such as app interactivity; 3) Support System, including 6 constructs, such as decision support or behavior change support features; 4) Autonomy, including 3 constructs, such as app accessibility in terms of cost; and 5) Safety, Privacy, and Trust, including 3 constructs, such as app safety for adolescent use.

View Figure

• Download as Powerpoint
• Download Figure

Discussion

This study found that adolescents prefer mHealth apps that are customizable, offer peer support through social media, sustain engagement via gamification, and support the ability to visualize health trends via simplified graphs. In terms of rating criteria for adolescent mHealth apps, the degree of ease of use, perceived usefulness, customizability, efficiency, appeal, privacy, credibility, and interactivity are commonly reported as rating criteria in studies that design, develop, or evaluate mHealth apps intended for adolescent use. Five dimensions emerged from the qualitative synthesis of adolescent preferences for mHealth app features and quality rating criteria. The dimensions include Technical Quality, Engagement, Support System, Autonomy, and Safety, Privacy, and Trust. Technical Quality includes quality dimensions pertinent to app usability. Engagement reflects the ability to sustain usage of the app due to the presence of positive features such as gamification. Support System includes mechanisms to aid users in different aspects relevant to adolescent mHealth app usage. Autonomy refers to the app being free of cost and the extent to which adolescent autonomy is enabled. For example, adolescent autonomy could be enabled by giving adolescents the liberty to share information with peers and healthcare providers, engage in an advocated behavior, or change the look and feel of the app without continual parental oversight. Safety, Privacy, and Trust describes the use of features such as a discreet design or password to protect users from negative events such as cyberbullying and social stigma. This dimension also covers credibility in terms of the app's visual appearance and information communicated via the app appearing trustworthy.

Adolescent preferences informed the theoretical framework proposed for the evaluation of mHealth apps targeted to adolescents in this study. Previous studies support our finding that adolescents prefer mHealth apps to be customizable18,28,29 and include social networking,30,31 gamification,18 charts and tracking,32 multimedia,33 and treatment reminders.18,28 Our findings on adolescents' need for social interaction features corroborates the findings of previous studies that reported that adolescents prefer their friends to be involved in self-management mHealth apps. For example, Roberts et al30 found that adolescents expressed a need to video-chat with friends, connect with friends from school, and create a communication network via social media within mHealth apps. Similarly, Bendixen (2017) reported that adolescents expressed the need for social media to provide a social support system in apps developed for adolescents.34 Additionally, previous studies support our findings on adolescents' preferences for concise and simple displays of data and textual information, especially in displays of tracked monitoring parameters for health conditions.18 App information and displays should also appear credible. For example, Nightingale et al35 reported accuracy and trustworthiness of app information as a desirable component of mHealth apps targeted to children and adolescents. This supports our findings that adolescents prefer mHealth apps that appear credible in terms of the visual design and tone or language used. Young adults age 18 to 30 years have also shown similar preferences to adolescents in this regard, reporting reluctance to read text notifications that did not appear credible.36 Finally, adolescents' need for autonomy in terms of the choice to use an mHealth app, sharing personal information within the app, and feeling in control of their health condition when using the app is another adolescent need confirmed by previous studies.29,37

Common mHealth app preferences between adolescents and adults include features that facilitate social networking, communication with healthcare providers, and customizability.38 Although both adolescents and adults express a need for social networking capabilities in mHealth apps, our findings show that adolescents prefer networking with their peers, or people of a similar age group, whereas adults prefer people with similar conditions, and trusted family and friends.38,39 Further, although both adults and adolescents prefer mHealth apps to include features enabling communication with healthcare providers, our study found that adolescents were primarily interested in providers' ability to view health data for monitoring purposes and the ability to ask questions and receive feedback from providers.19,20,24 Conversely, adults may prefer more advanced connections with the healthcare system, such as automation for prescription orders and refill processes, in addition to direct communication with healthcare providers.38 Adolescents may not see the need for these types of automated features because their parents or guardians typically obtain and refill prescriptions on their behalf. In addition to features facilitating communication with providers, adolescents prefer features that facilitate communication of data with parents and peers for informational purposes.8

A difference between adolescents' and adults' preferences in the design of mHealth apps is adolescents' need for features that support decision-making. This may include app suggestions for health condition-related actions based on tracked trends in monitoring parameters.8,25 Also, although both adolescents and adults have concerns about information privacy when using mHealth apps, the context may be different. For instance, cyberbullying may be a unique privacy concern for adolescents. Additionally, while our results indicate that adolescents prefer mHealth apps to include gamification components, such as the ability to earn points based on app use, electronic badges, and interactive quizzes, recent studies on adults' app preferences indicate that adults are less enthusiastic about gamification compared with adolescents, including the use of prizes, rewards, or competition to motivate health behavior.36,38

App rating criteria most often used by the included studies in adolescent populations are generally consistent with app quality rating criteria employed by app development and usability studies in adult populations. For instance, Birkhoff et al40 identified similar rating criteria themes in their app usability study, such as perceived usefulness, ability to track health, availability of reminders, and customizability. Communication, education, and socialization have also emerged as app rating criteria themes in a previous study involving adults.38 Further, several dimensions and constructs in the proposed framework align with constructs from established health behavior and/or technology use theories, theoretical frameworks, or conceptual models. For example, perceived usefulness and ease of use constructs are consistent with constructs from the Technology Acceptance Model and the Unified Theory of Acceptance and Use of Technology model.41,42

Tools for app quality assessments exist, including the App Quality Evaluation tool (AQEL)43 intended for nutritional apps and the Mobile App Rating Scale (MARS).44 The app rating criteria in AQEL consist of Appropriateness for Adults, App Functionality, Skill Building, Knowledge Building, Behavior Change Potential, and Appropriateness for Dietary Approaches to Stop Hypertension dimensions.45 The Knowledge Building, Skill Building, Appropriateness for Dietary Approaches to Stop Hypertension, and Behavior Change Potential dimensions are similar to constructs of the proposed framework's Support System dimension, including learning support, condition-specific support, and behavior change support. AQEL's Functionality dimension maps to a subset of the constructs within our proposed Technical Quality dimension, such as efficiency and ease of use. Similarly, AQEL's Appropriateness for Adults dimension achieves the same purpose as the Engagement dimension of our framework by ensuring that the app is targeted toward a specific age group. However, the AQEL framework does not address adolescent safety, privacy, and trust concerns, credibility, tailoring for adolescents in terms of the need for autonomy and social support, and may only be used for rating nutritional mHealth apps.

The MARS tool uses 4 overarching dimensions for evaluating the quality of mobile apps: Engagement, Functionality, Aesthetics, and Information. There are similarities between these dimensions and the proposed framework. For instance, the Engagement and Aesthetics dimensions in MARS are similar to the Engagement dimension in the proposed framework. Constructs of the MARS' Information dimension, such as credibility and accuracy of information, are constructs within the Safety, Privacy, and Trust dimension of the proposed framework. Further, the constructs measured under Functionality in MARS are similar to several constructs under Technical Quality in our framework. Despite these similarities, there are some differences between the proposed framework and MARS. The proposed framework is sensitive to apps created specifically for the adolescent population by capturing app features that enable adolescent autonomy, accessibility in terms of being free of cost, appeal to adolescents, and simplicity of textual and graphical data presentation. It also captures the need for multiple support systems in mHealth apps intended for adolescent use, including behavior change support, decision support, condition-specific support based on particular health conditions, social support, learning support, and institutional support. Finally, the proposed framework differs from MARS by addressing the issue of safety and privacy.

The Example of Medication Adherence. The proposed framework may be adapted to fit the needs of app developers and evaluators addressing a variety of adolescent health conditions and health behaviors. Here we describe an example of how the proposed framework might be adapted to fit general medication-taking behavior (non-condition-specific medication adherence) in adolescents. Dimensions of the framework that may be operationalized in a form befitting a medication adherence app for adolescents are described below. Technical Quality and Safety, Privacy, and Trust are generic and useful for app development or evaluation regardless of the health condition or behavior.

1. Engagement. The gamification construct may be adapted to transform behaviors such as medication taking or logging behaviors into fun activities for adolescents. Electronic badges or redeemable points may be awarded to users as they take and log their medications.

2. Support System. The 6 different types of support systems: behavior change, decision, condition-specific, social, learning, and institutional support may be adapted to support medication taking. First, behavior change support is critical to aid adolescents in forming and maintaining consistent medication-taking behavior. Relevant support may also include reminders to take medication and update medication logs (cues to action), feedback on medication-taking behavior displayed as simplified line graphs (self-regulation), and notifications about levels of adherence to the prescribed medication regimen (self-regulation). Second, decision support may take the form of suggested methods to become more adherent with medication taking if an adolescent's level of adherence is low. Third, a condition-specific support for medication adherence may include an app feature that enables adolescents to input the names and pictures of their medications, a medication logging feature, and a graphical display of medication taking or logging trends for specific medications or conditions. Fourth, the app may have a social networking component with the capability for adolescents to connect with individuals who also use the app for medication adherence purposes. Fifth, the app may offer links to or in-app educational resources for health conditions associated with users' medications, topics pertaining to the benefits of being adherent to medications, and information on typical medication dosages, side effects, drug-drug interactions, and contraindications. The final component of the support system, institutional support, may also be adapted. For instance, features that enable adolescents to share their medication log with school clinics and healthcare providers may be included to facilitate health monitoring or for incentivization purposes.

3. Autonomy. An important construct is adolescent autonomy. In this regard, questions to guide app developers or mHealth program planners are: 1) Does the app enable adolescents to feel in control of their medication-taking behavior? 2) Do adolescents have the capability to independently choose if adherence information is shared with others? 3) Is the app free of cost and available on multiple platforms such that adolescents can more easily access the app independently?

Study Limitations. This study is not without limitations. As the review is based on existing development and usability studies of mHealth apps intended for adolescent use, it is possible that salient preferences and app rating criteria themes may not have been captured by the original studies. However, due to the systematic procedure used in study selection and the relatively high quality of most of the retained studies, we have confidence that the preferences and rating criteria outlined here reflect those most salient to an adolescent population as represented in the current literature, and the proposed theoretical framework is suitable for evaluating mHealth apps intended for adolescent use. Also, this study did not include articles evaluating the usability of apps intended for adolescents with neurodevelopmental conditions, such as autism. This may limit the generalizability of findings to such populations.

Implications for Policy and Practice. This research adds new knowledge on app design concepts that may be adapted for the development of mHealth apps targeted to adolescent users. The dimensions in the proposed framework may also guide the evaluation of adolescent mHealth apps. In addition to including features that offer condition-specific support in mHealth apps targeted to adolescents, there is a need to incorporate features that offer other types of support pertinent to adolescents. This includes decision-making, behavior change, institutional, learning, and social support. Future studies should evaluate the most relevant types of support within mHealth apps that effectively promote healthy behaviors among adolescent users. It is also important for app developers to explore gamification within mHealth apps by using rewards like redeemable points and electronic badges, as well as app customizability and simplicity of textual and graphical information display. However, future studies should clarify which fonts, colors, and graphics are adolescent-friendly in order to facilitate effective implementation of gamification, customizability, and easily understood textual and graphical information displays in mHealth apps intended for adolescent use. Future research should also ascertain the ethical considerations for granting adolescents the autonomy to share health data with peers and healthcare providers via mHealth apps. Lastly, the proposed framework may be tested using exploratory and confirmatory factor analysis, and used to create a rating tool for adolescent mHealth apps after validation in samples of adolescents.

Conclusions

This paper describes adolescents' preferences for the design of mHealth apps, as well as the app rating criteria that have been used for this population. Adolescents value mHealth apps that are customizable and appealing due to the use of adolescent-friendly graphics, fonts, and colors. Apps that appear credible and use elements of gamification to support app use or the intended health behavior are also preferred. The incorporation of features described in this report may help in the development of mHealth apps that are preferred by adolescents, improving adolescents' engagement with and use of mHealth apps, which may in turn lead to improved self-management of chronic conditions and positive health outcomes for adolescents.