Virtual Pediatric Patient Activities With Randomized Scenarios as an Instructional Tool for Pharmacy Students

Introduction

Teaching methods focused on information dissemination and assessments focused on rote learning can assess the ability to recall recently presented information; however, they do not assess long-term learning or the ability to apply knowledge to a new situation.1 Ensuring that pharmacy students are able to integrate and apply foundational knowledge are important components of the American College of Pharmacy Education Standards and the Center for the Advancement of Pharmacy Education Outcomes.23 Active learning activities seek to engage students in learning versus allowing students to be attentive listeners, and may promote long-term learning and application-based clinical skill development to a greater degree.1

Many different types of active learning activities have been developed that teach students at various levels of the Bloom's taxonomy cognitive learning domain (the revised levels range from basic remembering and understanding to more complex learning involving analyzing, evaluating, and creating).14 A common active learning strategy employed in healthcare education is scenario-based learning. Instructional technologies have brought about new ways to simulate patient scenarios and create active learning environments in the classroom, usually referred to as virtual patients.5

Virtual patients used in healthcare education can be classified based on the technology utilized and desired competency outcome. For example, dynamic simulations and mixed realities can be used to develop procedural and basic skills while conversational characters or standardized patients are used to develop communication skills. The majority of virtual patients described in healthcare professional education involved the use of multimedia systems to promote clinical reasoning skills.6 There is much variability in the virtual patient scenario design and multimedia system capabilities, which can impact the effectiveness of the teaching method.67

Overall in health science education, virtual patients have made a positive impact on learning compared to no intervention, but the effects in comparison to other instructional methods were small and effects may depend on the type of virtual patients used.8 In pharmacy education, computer-aided instruction tools, mannequin-based patient simulations, and virtual patient simulations have been used. A recent review of technology in pharmacy education found that while computer-aided instruction revealed conflicting impact on learning, mannequin-based patients and virtual patient simulations had significant effects on learning.5 Analyses of virtual patient use in pharmacy schools have usually compared before and after knowledge assessments while comparisons to paper-based scenarios or other teaching methods have also not been done or did not show significant differences.9–12 The majority of virtual patient publications in the pharmacy and other healthcare fields have involved adult patient scenarios,5–7 but pediatric patients have unique medication-related needs that would impact the creation and implementation of virtual patients for pharmacy education.

Regarding clinical reasoning skill development in medical education, it has been suggested that practice with multiple different examples is needed to attain optimal knowledge transfer and long-term learning, or expertise.13–15 Yet, most published studies in medical education and all in pharmacy education have not analyzed the impact of repeated attempts of unique virtual patient scenarios on student learning.57 Additionally, all virtual patient scenarios in pharmacy education utilizing branched-outcome decision-making were based on adult patients.5 Thus, assessment of repeated attempts of pediatric virtual patients with branched-outcome decision-making and multiple scenarios is needed in pharmacy education. We theorized that having students repeatedly complete randomly presented similar, but unique, virtual patient scenarios could prevent rote memorization of a single scenario outcome and help students independently learn how to apply concepts to new complex patient scenarios.

We set to create two virtual pediatric patient activities (VPAs) that incorporated multiple disease states, branched-outcome decision-making based on text entry answers, and multiple scenarios and outcomes that were randomly presented to students. Our primary objectives to assess student learning from the VPAs were twofold. First, we aimed to determine if students were able to achieve a higher best score on the VPAs if they attempted them more times. Second, we set to compare the number of VPA attempts among students who correctly and incorrectly answered exam questions about patient scenarios that were similar, but unique from those in the VPAs. As a secondary objective, we sought to compare exam performance on the same questions after completion of the graded randomized VPAs to students who completed ungraded single paper-based scenarios.

Methods

Activity Development and Implementation. For the 2015 third professional year pharmacy students, 2 VPAs were created that randomly presented patient scenarios and utilized branched-outcome decision-making based logic, as shown in Figure 1. Based on previously described virtual patient classifications, the VPAs would fall under the classification of interactive patient scenarios seeking to improve clinical reasoning.6 The VPAs were created using the Qualtrics (Qualtrics, LLC, 2015, Provo, UT) electronic survey program. This program was chosen over others because it was available at the Virginia Commonwealth University School of Pharmacy, had robust data collecting capabilities, incorporated multiple question formats, could create scores based on responses, could randomize questions, and allowed for the use of branched-outcomes based on answers from multiple choice and/or text entry questions. The students were already taught the basics of the disease states and treatments included in the VPAs in previous courses. The VPAs were more complex than what the student had previously been exposed to as they combined multiple disease states and required students to identify relationships between the disease states in order to determine the optimal treatment plan.

View Figure

• Download as Powerpoint
• Download Figure

The asthma VPA could randomly have been 3 different scenarios, each having minor differences regarding subjective and objective information that affected the optimal therapeutic plan (Figure 1 and Table 1). Throughout the VPA, students assessed the patient's disease states and decided whether to continue, discontinue, or change medications. For the meningitis VPA, students were randomly presented 3 different sets of possible orders for verification. Students could only proceed to a second scenario of the meningitis VPA (with 3 additional randomized scenario outcomes) if the first scenario was correctly completed (Figure 1 and Table 1).

Table 1. Scoring and Detailed Description of Virtual Patient Activities

View Fullscreen

Both VPAs involved a combination of multiple choice, multiple selection, matching, and text entry questions as described in Table 1 (e.g., for drug names, organisms, dosing, frequencies, and start times). The text entry questions allowed for students to create plans versus selecting a plan from a predetermined list, which is the highest level of the Bloom's taxonomy cognitive learning domain (i.e., creation).4 Students were encouraged to double check spelling or copy and paste organisms and drug names into the system. Additionally, to accommodate for minor spelling mistakes, “if contains” and “if does not contain” statements were used and included 1 or more fragments of the whole name in order to differentiate from other medications (e.g., “if contains ceftr” for ceftriaxone). Generic medication names only were allowed for entry into the system and capitalization did not affect the decision logic. Ranges for dosages and times were also utilized to accommodate for rounding (within 10% of minimum and maximum, and within 30 minutes of the correct time). Students were given written directions on how to complete the VPAs 6 days prior to class (including suggestions for references to use). The VPAs were then made available via a Web link for students to attempt as many times as they wanted before class individually.

The students also received a total score after each VPA attempt, based on responses entered or selected and accounted for the best (full credit) versus alternative treatment plans (partial credit). Details regarding activity scoring are provided in Table 1. Any notes, electronic references, paper references, or guidelines could have been used to help complete the VPAs. Patient outcomes were used as feedback within the VPAs to suggest correct or incorrect patient plans; however, correct answers were not given to the students. For example, if a dosage of gentamicin was too high, the student received feedback that the patient developed acute kidney injury instead of being told the correct dosage of gentamicin.

Students were required to have attempted each VPA at least once before class in order to receive credit for the activity. During an in-class session, students worked in 23 groups (with 5–6 students in each group) and were allowed unlimited VPA attempts of the same 2 VPAs completed before class. The highest score the group received counted as a small portion of their group activity grade for the course. The second part of the activity was a didactic lecture that reviewed the VPA scenarios and added clinical pearl information. After class, the same VPAs were reopened and students were allowed unlimited attempts for exam preparation.

In 2014, two of the possible scenarios incorporated into the VPAs (1 asthma and 1 meningitis) were used in paper form as a learning activity. Individually before class, students completed a series of questions (related to their therapeutic plans and also the same questions used in the VPAs in 2015) comprising important scenario aspects. Then, they worked in groups in class to answer the same questions and create a plan. This activity was followed by the same lecture (by the same faculty member) as 2015 except for minor changes due to the additional scenario possibilities in the VPAs. The activity did not count for a grade.

Each year, 8 scenario-based exam questions were asked to assess student learning of the material from this activity. Seven of the questions were nearly identical between years. Three of the questions each year were questions about alternative scenarios to the randomized portions of the VPAs in 2015 (e.g., verification of the accuracy of medication orders). These required students to apply concepts learned during the activity to a similar, but unique scenario. The other 4 questions were related to information covered in the VPAs and subsequent didactic portion, but were not alternative scenarios and thus could have been answered by scenario or information recall.

Data Collection. For descriptive purposes, the time it took to complete each VPA attempt was recorded. As Qualtrics is a Web-based survey program, Web browsers could be closed without submitting the attempt (and after ceasing work on the VPA), but the attempt timer would not stop until the VPA was submitted or the instructor closed the survey. Thus, any attempt recorded as taking over 2 hours to complete was removed from the time analysis, but still included as an attempt.

To assess learning among individuals after repeated VPA attempts, the number of individual attempts (before class and after class) was collected for each VPA and divided into approximate quartiles, or thirds, when possible. The best score achieved individually was also collected. Group best scores were not utilized for analysis as the relationship between number of attempts and group score may be biased by one individual's effort for the group. Additionally, a group may not have continued to try the VPAs once they achieved a perfect or what the group felt was a “good score.”

For analysis of application abilities on exam questions, students were divided into 2 groups based on if they correctly or incorrectly answered the exam questions that were alternate scenarios of the randomized portions of the VPAs. We collected the number of times each individual student attempted the VPAs before class, in class as part of a group, and after class. For comparison between groups for the fourth meningitis question related to the second scenario of the meningitis VPA, the number of attempts was based on their attempts at the second scenario of the VPA. The grade point average (GPA) of each student at the start of the course was also collected as a potential confounding variable between groups. Individual exam question scores were also collected for the 2014 class to allow for comparison between years.

Exam questions were independently reviewed by 3 faculty members (authors 1 and 3, in addition to a faculty member not directly involved in the project) and categorized using the revised Bloom's taxonomy.4 Final categorization was based on agreement between at least 2 of the faculty, or the middle category if all 3 were different. The Virginia Commonwealth University Institutional Review Board approved this analysis as an exempt study.

Statistical Analysis. Descriptive statistics were presented as means with 95% confidence intervals or medians with interquartile ranges based on data normality. Mean or median VPA attempts, VPA scores, and GPA for groups were compared using the independent T-test, Mann-Whitney U test, or Kruskal-Wallis test as appropriate based on data normality and the number of groups being compared. Categorical comparisons were done using the χ² test. All statistical analyses were performed using IBM SPSS Statistics version 23 (IBM Corp, 2015, Armonk, NY). Significance was set a-priori at p < 0.05.

Results

Out of the 132 students in the 2015 class, 1 student did not attempt either VPA before class and 3 students did not attend class. Every student attempted the asthma VPA at least once before the exam, but 1 student did not attempt the meningitis VPA. In summation, there were 627 VPA attempts before class (246 attempts, 1.9 per student, for the asthma VPA, and 381, 2.9 per student, for the meningitis VPA). On average, students finished an asthma attempt in 21.6 minutes before class, and a meningitis attempt in 17.4 minutes, which after adjustment for the number of attempts per student totals 91.1 minutes of preclass preparation. During the 75-minute in class activity, 340 VPA attempts were completed as a group (153 attempts, 6.7 per group, for the asthma VPA, and 187, 8.1 per group, for the meningitis VPA). Groups on average completed an asthma attempt in 7.1 minutes and a meningitis attempt in 12.5 minutes. Some groups may have submitted multiple VPAs at 1 time. After the class occurred, 50 students attempted at least 1 of the VPAs, for a total of 134 attempts in preparation for the exam (33 attempts for the asthma VPA, 1.3 per student, and 101 for the meningitis VPA, 2.1 per student). The VPAs were finished after class in 14.2 minutes and 22.4 minutes, for asthma and meningitis, respectively.

Students who individually completed the VPAs more times achieved a higher best score on both the asthma and meningitis VPAs (Figures 2 and 3). The student who correctly answered the meningitis exam question related to the first randomized portion of the meningitis VPA (verifying medication orders entered) attempted the VPA more times in class as a group (Table 2). There were no differences between groups for the fourth meningitis exam question, and GPA was the only difference for the related asthma exam question (Table 2).

View Figure

• Download as Powerpoint
• Download Figure

View Figure

• Download as Powerpoint
• Download Figure

Table 2. Comparison^* of Virtual Patient Activity Attempts Based on Related Exam Question^† Correctness

View Fullscreen

Students' exam scores in 2015 were significantly higher than in 2014 for the 7 questions overall and for 3 individual questions, although the score for the fourth meningitis question was significantly lower in 2015 (Table 3). This question was related to the second part of the meningitis VPA, which some students may not have seen if they did not make it to the second part of the VPA. After removing students who did not get to the second part of the VPA (either individually or as a group), the proportion of students who got the question correct increased to 79.4% in 2015 (compared to 86.0% in 2014, p = 0.181).

Table 3. Exam Question Performance Comparison Between Years

View Fullscreen

Discussion

The created pediatric VPAs, with randomized scenarios and branched-outcome decision-making, were learning activities that achieved participation from nearly every student in 2015. These data suggest that students achieved better VPA scores with repeated VPA attempts. Impact of the number of group attempts on exam performance was noted when the exam question was related to an aspect of the VPA that forced students to correctly complete the scenarios to continue with the VPA (e.g., a more severe consequence for an incorrect action). Our study also suggests that the graded VPAs improved exam score performance compared to a single ungraded paper-based scenario.

To the knowledge of the authors, this study was the first in the pharmacy literature to analyze the impact of repeated VPA attempts on student learning. Since feedback given in the VPAs did not provide the correct answer for the given scenario, repeated attempts required students to actively elucidate the consequences of incorrect decisions after each VPA attempt. The improvement in best VPA score achieved based on the number of VPA attempts (Figures 2 and 3) suggested students who completed the VPAs more were able to independently learn how to appropriately design plans for the VPA scenarios. This independent active learning was then reinforced by group VPA attempts and didactic review of the scenarios. Our data agree with observations in the medical literature suggesting that repeated exposures to different scenarios are an effective form of learning.13–15 In the asthma VPA, students scored better overall on attempts, likely due to the inability to complete meningitis VPA if the first scenario was not appropriately completed. The difficulty with this task was not anticipated and may require adjustments in scoring to make assigning a VPA grade easier.

Students who correctly answered the meningitis exam question related to the first randomized scenario (meningitis exam question 3) had attempted the VPA more times in class. This suggested that students may have better learned the application skills required for this part of the meningitis VPA from repeated group attempts of the different randomized scenarios. Our study is in agreement with the medical education literature suggesting that long-term learning is better achieved with practice or repeated exposures to different scenarios.13–15 Our data also support the utility of group work in facilitating learning during VPAs. A meta-analysis of computerized virtual patients in healthcare education has also suggested that working in groups or teams is an important aspect of VPAs.8 The first part of the meningitis VPA and related exam question (meningitis exam question 3) were directly related to student's ability to identify and correct errors in prescribed medications, a fundamental skill required of a pharmacist providing care for a pediatric patient. A future direction could be to determine if presentation of randomized medication orders could help train pharmacists to identify errors in medication orders presented for pediatric patients. There was not a significant difference for the third meningitis exam question between years (e.g., paper scenarios versus virtual VPAs) (Table 3). A larger sample size of students with numerous attempts may be needed to see a significant difference in this comparison.

Students who correctly answered the meningitis question related to the second part of the meningitis VPA and the related asthma question did not have a different number of VPA attempts, while prior GPA was generally higher in those who correctly answered the questions (Table 2). This suggests that if students are not forced to correctly complete the activity, GPA likely has a greater association than VPA attempts. For the asthma VPA-related exam questions, performance improved between years on the fourth asthma question related to the randomized aspect of asthma VPA. This suggested that doing a VPA was sufficient to learn how to apply the required concept and the presentation of randomized scenarios within the VPA may not have impacted their learning. Some students were not able to correctly complete the first part of the meningitis VPA, and consequently did not experience the entire meningitis VPA (except as part of the didactic discussion). These students performed worse between years on the fourth meningitis exam question related to VPA scenario randomization. This may suggest that only reviewing material didactically achieved less learning than the VPAs and the paper-based scenarios. To help mitigate this problem in the future, it would be beneficial to design the activity so those students would experience the whole VPA at some point before an exam.

The exam questions were difficult questions, with all except 1 assessing students learning in high Bloom's taxonomy levels. The questions were based on complex patient scenarios with many different aspects. They required students to identify and analyze pertinent information, and to apply the information to make decisions regarding patient care. Lower than average student performance was to be expected (especially on the fourth asthma question and third meningitis questions). Notably, students scored better on the same 7 exam questions in 2015 compared to 2014 (Table 3). This suggested that students were more able to apply the knowledge they gained to difficult scenario-based questions, potentially due to the completion of the VPAs. The individual questions that students performed significantly better on in 2015 included questions categorized as understand, analyze, and evaluate based on Bloom's taxonomy.4 This signified that the VPAs taught students skills at various levels of learning. While these results are promising, it is still not certain whether the use of virtual patients, randomized virtual scenarios, or grading the assignment (likely increasing participation in the learning activity) helped improve scores between years.

This study is unique in the virtual patient pharmacy education literature. While previous studies have used branched-outcome decision-making for virtual patients, the scenarios did not include pediatric patients, utilize text entry questions, or analyze randomization of multiple scenarios.912 A previous study did randomize scenarios for a 2-year virtual patient project, but this study did not analyze quantitative learning associated with the randomization.16 In addition, previous studies have not analyzed the association between the number of student attempts and the number of correct answers on an examination. The benefit of using virtual patients in pediatric education has been previously studied in non-pharmacy fields. Among nurse practitioner students, it was found that the implementation of 2 multimedia virtual pediatric patient scenarios resulted in a significant increase in knowledge and comfort level in regards to working with pediatric patients with developmental disabilities.17 Likewise, it has been demonstrated that interactive virtual patients improved student dentists' competency in providing care to children with disabilities.18 However, these studies did not have randomized scenarios and used before and after basic knowledge assessments as an outcome. In medical education, a VPA did use an adapting scenario and analyzed exam performance as an outcome, but did not find a difference between students who did and did not complete the VPA.19 Virtual patient use in the medical field has shown variations in effectiveness potentially dependent on educational design, desired outcomes, time allotted, and the number of VPAs completed.7

We are the first to use the Qualtrics survey software as a framework for VPAs.5 The use of Qualtrics, with branched-outcome logic and data collection capabilities, allowed for more complex scenarios and facilitated active student learning in higher levels of the Bloom's taxonomy cognitive learning domain.4 However, the user interface did not aesthetically mimic an electronic health record and in the future we would like to make the scenarios more graphically appealing, real-life, and easier to use. Based on our findings, other similar VPAs have been constructed and utilized for patient scenario based learning in other learning sessions.

There are limitations to our analysis of the VPAs. It is possible that the lecturer could have taught the material differently the second year, although this possibility was minimized as the same lecturer taught the content both years. Another difference between the 2 years could have been that some of the 2014 students were on distant campuses, whereas the 2015 students were all on the same campus. However, previous studies have not shown a difference between distance and on-site learning for scenario based learning activities.20 For the yearly comparison analysis, it is important to note that the assignment was not graded in 2014, but the VPAs did count for a grade in 2015 due to required changes in the course grading, though this was a secondary objective of the study. Comparison of a graded single scenario to randomly presented virtual scenarios may still require further analysis. We did not analyze student perceptions of the assignment, and this could be a focus of future studies. Finally, the VPAs were only implemented at 1 institution using the Qualtrics program. Thus, these results may not apply to other institutions or other software with randomization and branched-outcome logic capabilities.

Conclusions

Two sets of randomized, virtual, branched-outcome decision-making pediatric patient VPAs were used as active learning activities to teach students how to make medication-related decisions for complicated pediatric patients. Students who completed the VPAs more times were able to achieve greater individual best scores. Students who correctly answered exam questions had a higher number of group VPA attempts only when continuation of the VPAs required correct randomized scenario completion. The use of graded VPAs versus ungraded paper-based scenarios led to higher student exam scores on questions requiring students to apply knowledge and skills to a new scenario. A virtual pediatric patient activity with randomized scenarios may be a useful tool for facilitating student learning at high levels of the Bloom's taxonomy cognitive learning domain and assisting student pharmacists with clinical skills development.