Videoconference Administration of the Western Aphasia Battery

Videoconference administration of the Western Aphasia Battery - Revised: Feasibility and Validity Abstract Purpose: There is rapid growth of telepractice in both clinical and research settings; however, the literature validating translation of traditional methods of assessments and interventions to valid remote videoconference administrations is limited. This is especially true in the field of speech-language pathology where assessments of language and communication can be easily conducted via remote administration. The aim of this study was to validate videoconference administration of the Western Aphasia Battery – Revised (Kertez, 2007). Method: Twenty adults with chronic aphasia completed the assessment both in person and via videoconference with the order counterbalanced across administrations. Specific modifications to select WAB-R subtests were made to accommodate interaction by computer and internet. Results: Results revealed that the two methods of administration were highly correlated and showed no difference in domain scores. Additionally, most patients endorsed being mostly or very satisfied with the videoconference administration. Conclusions: These findings suggest that administration of the WAB-R in person and via videoconference may be used interchangeably in this patient population. Modifications and guidelines are provided to assure reproducibility and access to other clinicians and scientists interested in remote administration of the WAB-R. 2 Introduction Continued technological advancement is finding its way into many fields, one of them being healthcare. Healthcare providers are increasingly implementing the use of telehealth, which is defined as “technology-enabled health and care management and delivery systems that extend capacity and access” (“Telehealth Basics,” 2019). The American Hospital Association reports that in 2017, 76% of hospitals in the United States alone had implemented some form of telepractice (“Fact Sheet: Telehealth,” 2019). Additionally, a recent study by the National Business Group of Health reported that 96% of employers in the United States will be providing medical coverage for telehealth in states where it is allowed (“NBGH Press Release,” 2017). According to the American Telemedicine Association, a recent push for telepractice is underscored by the need for improved access to care, cost efficiency, improved quality, and patient demand (“Telehealth Basics,” 2019). Of these, accessibility to care is especially significant for those patients who otherwise may not have medical resources available to them. Telemedicine and telepractice now enable patients living with illnesses in remote areas to receive the same services as those who live in large metropolitan centers with easy access to high-quality care. As this practice continues to grow, it is imperative to validate the delivery of assessment and treatment to ensure that the same level of care is provided to patients who are evaluated remotely as to those evaluated in the clinic (i.e. in person). The field of speech-language pathology is in the process of expanding its telepractice reach and capabilities (Cherney & Van Vuuren, 2012; Duffy, Werven, & Aronson, 1997; Theodoros, Hill, Russell, Ward, & Wootton, 2008). In recent years, both teleassessment (i.e. remote assessment of speech, language and cognitive function) and telerehabilitation (i.e. 3 treatment of speech, language and cognitive impairments) services have been offered clinically and in research settings by speech-language pathologists (SLPs) (Brennan et al., 2010; Hall, Boisvert, & Steele, 2013). Specifically, according to a 2016 American Speech-LanguageHearing Association (ASHA) survey that was completed by 476 SLPs, 64% endorsed providing services using telepractice (American Speech-Language-Hearing Association, 2016). The survey followed up with a question regarding how the SLPs utilized telepractice and found that 37.6% of SLPs used telepractice for screenings, 60.7% used telepractice for assessment, and 96.4% used it for treatment. Although there is clearly an interest in providing telepractice in the realm of speech, language, and hearing sciences, there is a lack of literature on validated telepracticebased assessments and intervention programs. Further, while there are countless speech, language and cognitive assessments, translation of these traditional methods of assessments to valid remote videoconference administrations is limited. To our knowledge, only two assessments for aphasia (both short forms) have been validated for videoconference administration: the Boston Naming Test (BNT-2 Short Form) (Kaplan, Goodglass, & Weintraub, 2001) and the Boston Diagnostic Aphasia Examination (BDAE-3 Short Form) (Goodglass, Kaplan, & Barresi, 2001) (Hill, Theodoros, Russell, Ward, & Wootton, 2009; Palsbo, 2007; Theodoros et al., 2008). Hill and colleagues conducted a study with 32 adults with acquired aphasia and reported that severity of aphasia did not affect the accuracy of telepractice assessment of the BDAE-3 Short Form except in assessing naming and paraphasias. Statistically significant differences in rating scale scores in these domains were found for telerehabilitation assessment vs. face-to-face assessment for severely impaired individuals only (Hill et al., 2009). 4 Recently, a group of experts in aphasia reached a consensus to consider the Western Aphasia Battery - Revised (WAB-R) (Kertez, 2007) as the core outcome measure for language impairment in aphasia (Wallace et al., 2019). The WAB-R is a comprehensive test battery for linguistic skills along with several non-linguistic components that provides three scores: Aphasia Quotient (AQ), Language Quotient (LQ), and Cortical Quotient (CQ). The AQ is often used as a measure of overall aphasia severity; LQ is a measure of spoken and written language performance; and CQ combines linguistic and non-linguistic performance. Given the diagnostic capabilities of this test and its status as a core outcome measure for aphasia rehabilitation per expert consensus, it is important to validate teleassessment of the WAB-R in adults with aphasia. Yet, there are several obstacles to validating remote administration of an assessment like the WAB-R. First, the WAB-R does not lend itself to easy web-based administration as there are objects to manipulate and several subtests require pointing or gesturing. In addition, computer assessments in individuals with post-stroke aphasia oftentimes require the help of a caregiver due to reduced comprehension skills in this population. The goal of this study was to establish the feasibility of teleassessment of the WAB-R and to determine if videoconference administration was comparable to in-person administration. Furthermore, we sought to determine whether people with aphasia (PWA) preferred in-person administration or videoconference administration, or if they felt similarly about both methods of assessment. Finally, like Hill et al. (2009) we examined whether there were differences in administration and scoring that were influenced by aphasia severity. Method A total of 20 adults with chronic acquired aphasia (mean age = 55 years old, mean years of education = 16) were recruited from the New England area (see Table 1 for demographics). 5 All participants were in the chronic stage of recovery (i.e. at least 6 months post injury) and did not present with any significant pre-morbid psychiatric or neurologic illness. 12/20 participants reported being involved in aphasia community groups within the six months prior to the study. 12/20 participants reported being involved in individual speech therapy within the six months prior to the study. Five of those participants has completed individual treatment prior to study onset. For the remaining seven, two were actively involved in individual treatment and for five participants it is not known whether they were actively involved in treatment. All participants provided informed consent according to the Boston University Institutional Review Board. PLEASE INSERT TABLE 1 ABOUT HERE Set-up At the onset of the study, the researchers inquired about the type of computer participants had at home and whether it had the required components for the study such as video camera, microphone, and speakers. If the participant did not have a computer that was compatible with videoconferencing, a laptop was lent to him or her in order to complete the videoconference portion of the study. An important aspect of this study was to maintain ecological validity; therefore, participants were encouraged to use their own technology if possible. In addition to computer equipment, participants were asked whether they had the following materials at home: white printer paper, a comb, a book, and a pen. If they were missing any of these testing materials, materials were provided along with four Koh’s blocks. Prior to assessment, participants completed a questionnaire regarding their current technology use. Questions were open-ended to allow participants to comment on the different types of devices they used at the time of the study and in what capacity (See Supplemental Figure 1). Lastly, researchers asked caregivers to be readily available to assist during assessment administration if necessary. The 6 researchers provided consistent guidelines as to how much the caregivers should assist (i.e. do not provide feedback on performance, repeat instructions given by the clinician, or allow the use of notes or aids) (See Supplemental Figure 2 for full description of verbal instructions provided at time of testing). Assessment All participants completed both in-person and videoconference testing with the WAB-R. The order of administration (in-person or videoconference) was counterbalanced across participants, with 10 participants completing videoconference testing first and 10 participants completing inperson testing first. Initial counterbalance order was intended to be every other participant assigned to videoconference vs. in-person assessment. This counterbalance order was modified for convenience as needed based on study team and participant schedule demands. Tests were completed at an interval of 7-14 days (See Figure 1). All in-person assessments were completed either at the Boston University Aphasia Research Laboratory or in the participant’s home with a trained research assistant. Videoconference administrations were completed using either the GoToMeeting or Zoom platforms (GoToMeeting, 2018; Zoom, 2018). One important aspect of videoconferencing is to ensure that participant privacy and confidentiality is enforced. In this project, two platforms for videoconferencing were utilized, GoToMeeting and Zoom. Both have specific security settings to ensure patient confidentiality. Although both platforms worked well for the current project, the Zoom platform may be a better choice for most clinicians and researchers. It allows for easier sharing of documents with participants as well as meets all HIPAA requirements. 7 As much as possible, videoconference assessment was completed with participants in their home environment to simulate clinical telepractice. For the three participants who preferred to complete the videoconference testing in the clinic, they were set up with a laptop in a separate room and were not provided assistance with the testing by the researchers. PLEASE INSERT FIGURE 1 ABOUT HERE Several modifications were made for the videoconference administration of the WAB-R (see Figure 2). First, all stimuli were scanned and uploaded as a digital .pdf file. These files were shared with participants (only during test administration), allowing them to point to the stimuli with their mouse or their finger if using a touch screen device. Additional stimuli and task specific modifications that were made are detailed in Appendix A. PLEASE INSERT FIGURE 2 ABOUT HERE After assessment At the end of the experiment, participants once again completed a questionnaire based on Hill and colleagues’ work rating the following: overall satisfaction, audio and visual quality, level of comfort, whether they would participate in videoconference assessment again, and whether they were equally satisfied with in-person and videoconference assessment (Hill et al., 2009). Assessment Fidelity and Reliability Several steps were taken to promote assessment fidelity prior to the study, to minimize provider “drift” during the study, and to verify fidelity after test administrations. In addition to emphasizing the importance of assessment fidelity, Richardson and colleagues outline details to be considered and reported during aphasia assessments including assessor qualifications, 8 assessor and rater training, assessment delivery, scoring reliability, and assessor blinding, which are discussed below (Richardson, Dalton Hudspeth, Shafer, & Patterson, 2016). Three trained research team members completed administrations of the WAB-R. Administrator 1 had six years of experience administering neuropsychological tests in a research context, including two years of experience administering the WAB-R. Administrators 2 and 3 were speech-language pathologists with six and two years of experience, respectively. Prior to beginning the study, administrators reviewed the test together and clarified scoring questions. Administrators referred to the WAB-R scoring booklet and manual for scoring questions as well as a team-developed document of clarifications for common scenarios not specified in the WABR manual (See Appendices B and C). All WAB-R administrations, both in-person and videoconference, were video-recorded to allow for fidelity and reliability review. Administrators 1 and 2 engaged in a blinded calibration process prior to initiating the study. Administrator 1 blindly rescored two of Administrator 2’s evaluations from video-recording and Administrator 2 blindly reviewed three of Administrator 1’s evaluations using the video-recordings. The administrators then reviewed both sets of scores for all five administrations, discussed discrepancies, and reached a consensus regarding scoring procedures for subsequent evaluations (see Table 4 for order of administrations and administrators). After calibration, only scores from the clinician performing the evaluation were used in this project. PLEASE INSERT TABLE 4 ABOUT HERE Although blinded assessment was not possible for this study, for 17/20 participants, different assessors administered in-person vs. videoconference assessment to reduce 9 administrator bias in re-assessment. Each examiner for the second administration did not review assessment scores for the first administration. Treatment fidelity and reliability were assessed by an independent rater not involved in testing. The research team developed a 32-item fidelity checklist with binary responses (See Appendix D). Mid-way through the research study, this assessor reviewed videos for WAB-R administrations for four participants from two administrators using the fidelity checklist (eight total administrations, comprising 20% of the final number of administrations). Following this review, the assessor discussed discrepancies with the administrators. This was done to minimize drift in assessment procedures as the study progressed. In addition, post-hoc review of participant scores that differed more than 5 points in AQ or that showed a change in aphasia type between administrations was completed to determine whether discrepancies in assessment administration or scoring may have contributed to score changes. Statistics Intraclass correlations (ICCs) and paired-samples t-tests were calculated to compare the WAB-R scores (AQ, LQ, CQ) between videoconference and in-person administrations. Analyses were performed using IBM SPSS Statistics 22 and analyses were considered significant at a conservative alpha-level of .01. ICCs were conducted with a two-way mixed effects model with absolute agreement. Mean estimates along with 99% confidence intervals (CI) are reported for each ICC. Interpretations are as follows: <0.50, poor; between 0.50 and 0.75, fair; between 0.75 and 0.90, good; above 0.90, excellent (Koo & Li, 2016; Perinetti, 2018). For reliability assessment, the independent rater rescored eight WAB-R administrations for two administrators without reviewing original scores (20% of total administrations). Domain 10 subscores across the eight domains of the test (Spontaneous Speech; Auditory Verbal Comprehension; Repetition; Naming and Word Finding; Reading; Writing; Apraxia; and Constructional, Visuospatial and Calculation) were tabulated and z-scores were generated. ICCs were conducted on a summary measure for these z-scores to compare scores given by the initial administrator to scores given by the independent rater with a two-way random effects model with absolute agreement. Comparisons were omitted for six missing data points in which the independent rater was unable to complete rescoring (e.g., video-recording malfunction). Results A high degree of reliability was found between in-person and videoconference administrations for measures of AQ, LQ, and CQ. The ICCs for the three WAB measures were all excellent, with reliability being .989 (99% CI = .964-.997; p = <0.001) for AQ, 0.993 (99% CI = 0.978-0.998); p = <0.001) for LQ, and 0.993 (99% CI = 0.978-0.998; p = <0.001) for CQ (see Table 2). Paired samples t-tests for the three domain scores showed no significant differences between in-person and videoconference administration for AQ (t(19)= -.856, p =.403), LQ (t(19) = -0.859, p = 0.401), and CQ (t(19) = -.073, p = 0.943) (see Table 2). Finally, the same analyses (both ICCs and paired samples t-test) were performed to analyze the differences between first and second administrations (regardless of whether these were in-person or videoconference administrations). All ICCs were found to be excellent (i.e. >0.9) and all t-tests were non-significant at an alpha-level of .01. PLEASE INSERT TABLE 2 ABOUT HERE Fidelity and Reliability 11 Assessment fidelity as measured by percentage of items on the checklist with a score of 1/1 was 94% for in-person administrations and 92% for videoconference administrations. Assessment fidelity was 91% for Administrator 1 and 95% for Administrator 2. In-person vs. videoconference and Administrator 1 vs. Administrator 2 comparisons indicate a high degree of assessment fidelity at the mid-point of the study. For the limited number of evaluated items that did not receive a score of 1/1 in the checklist, no clear patterns of administrator error emerged. In terms of reliability, ICCs were conducted comparing initial evaluator score with independent rater score showing a correlation coefficient of .991 (99% CI = .982 - .995; p<.001), indicating a high degree of inter-rater reliability. Satisfaction Survey The survey results revealed that 85% of participants did not show a preference for one form of administration over the other (see Figure 3). Of the three individuals who endorsed being more satisfied with in-person assessment, two had reported not feeling comfortable with technology prior to the study. One stated she had no concerns with in-person testing and that videoconference testing added an additional level of difficulty. Eighty percent of PWA reported being either mostly or very satisfied with the videoconference testing, fifteen percent reported being neutral about the administration method, and only one individual said she was less than satisfied with the videoconference administration. This was the same participant who reported it was easier for her to complete in-person testing. However, all individuals reported that they would be willing to complete teleassessment again. Additionally, participants endorsed good audio and visual quality (for more detailed information refer to Figure 2). PLEASE INSERT FIGURE 3 ABOUT HERE Individual patient analyses 12 Although we found robust results indicating that the two methods of WAB-R administration may be used interchangeably, there were some individual-level inconsistencies that should be highlighted (See Table 3 for individual scores). Of note, five participants (P8, P9, P14, P18, and P19) showed changes in aphasia classification per the WAB-R Classification Criteria between first and second administrations. In three of five cases, this was due to variation in the Repetition score across the boundary of 7/10 that distinguishes conduction from anomic aphasia (Participants 8, 18, and 19). In these cases, Repetition scores were not consistently higher in either videoconference or in-person assessment. In two of five cases, classification change was due to variation in the Auditory Verbal Comprehension score, higher in one instance for in-person assessment (Participant 14) and higher in the other for videoconference (Participant 9). Only one of these five participants falls in the group that showed greater than a 5-point change in AQ (Participant 19), showing that WAB-R classification can change while overall severity rating remains constant. It should also be noted that Participants 8, 9, and 18 reported being involved in individual treatment in the 6 months preceding the study and that this treatment had concluded prior to study onset. Participant 14 reported being involved in treatment in the 6 months preceding the study and it is not known whether this treatment was ongoing. Additionally, three participants presented with AQ or LQ changes greater than 5 points, a benchmark found to be a significant indicator of change (Gilmore, Dwyer, & Kiran, 2019). No clear pattern of change emerges amongst these three participants in terms of domain scores. Participant 2 showed an increase in AQ from 39.2 to 44.6 between first (videoconference) and second (in-person) administrations. In Participant 2, this difference appears to be largely driven by increases in Auditory Verbal Comprehension and Repetition. This participant was older (72 years old) and had reported hearing difficulty, which may have partially contributed to the 13 discrepancy in the scores between administrations, given that sound quality may have been degraded in the videoconference assessment. Of note, Participant 2 reported he had been involved in individual treatment in the six months preceding the study; it was unknown whether this treatment was ongoing. In addition, Participant 17 showed a decrease in AQ from 89.6 to 80.9 and a decrease in LQ from 90.7 to 84.6 between first (videoconference) and second (in-person) administrations. The AQ change was driven primarily by a three-point decrease in the fluency score between the first and second time points, with more complete sentences produced by the participant during the first assessment. Upon review of this administration, one possible explanation for the discrepancy may be timing of the administrator prompt for full sentences. During the first administration, the clinician prompted during the participant’s picture description, while the second clinician prompted prior to the response. The later prompting may have provided an opportunity to reflect on and modify performance, given that the participant produced more fluent sentences following the prompt provided mid-response. Lastly, Participant 19 showed an increase in AQ from 69.8 to 77.4 between first (videoconference) and second (in-person) administrations, with a slight increase across all domains. Given this participant’s limited experience in the testing environment (both clinically and in a research setting), increased comfort with the testing environment at second administration may have been a factor in improved performance. Discussion In this study, we examined the feasibility and validity of videoconference administration of the WAB-R in PWA. We found that participants with a variety of aphasia severities successfully completed videoconference assessment. Our participants exhibited a range of ages 14 from 26 to 75 years old, indicating that participants across the lifespan are able to complete videoconference testing. We also found that scores for the WAB-R administered in person as compared to via videoconference were highly correlated and therefore conclude that teleassessment is a reasonable option for this population. Finally, most participants endorsed being satisfied with videoconference assessment and would be willing to participate in future work involving this form of test administration. The validation of teleassessment in this population is particularly important in order to provide greater accessibility to healthcare. Nearly 180,000 Americans acquire aphasia annually, with the primary cause being stroke followed by secondary cause being traumatic brain injury (TBI) (NIDCD, 2019). Many of these individuals suffer from comorbid physical disability, with 80% of stroke patients experiencing post-stroke hemiparesis in the acute stage and more than 40% experiencing it chronically (Cramer et al., 1997). Due to significant physical disabilities, these individuals may have difficulty with independent transportation to medical appointments. Coupled with their communication difficulties, accessing high quality medical care may not always be attainable for this population. Videoconference assessment provides a viable alternative to travelling to a medical facility. In addition to making videoconference technology more easily accessible to PWA, it also needs to be made more accessible to clinicians. Validation studies such as this one are important to establish standardized guidelines for teleassessment in both clinical and research settings. A study conducted by Grogan-Johnson and colleagues (2015) surveyed academic programs that provide clinical training to graduate students in speech-language pathology. Their findings revealed that of the 92 academic programs with a university clinic in the sample, 24% of the programs used interactive videoconferencing to deliver speech and language services. 15 Additionally, they found that among programs that offer videoconferencing services, only 1-25% of graduate students receive direct clinical training in this type of administration (GroganJohnson, Meehan, McCormick, & Miller, 2015). This scarcity in training needs to be mitigated as more clinicians are offering videoconference services in their clinic without receiving training in their graduate courses on the proper administration of these assessments. We believe that by providing guidelines for administration, we can offer educational materials to clinicians utilizing this method of assessment. Additionally, we hope to share evidence supporting the efficacy of videoconference assessment for clinicians who may be considering using this method of assessment administration. There are several considerations that need to be mentioned. First, most participants who completed this study required some level of caregiver assistance. Most of the assistance was provided in the initial set-up stages of the experiment (i.e. opening the videoconference platform, turning on sound/microphone). Therefore, we recommend a caregiver be available to assist with teleassessment. At the same time, it is important to ensure the caregiver does not provide more support than set-up to ensure validity of the test results. Another issue to consider is participant mobility. Individuals with hemiparesis should be provided a computer mouse if needed to aid in selection of items during the test administration. Finally, this study did not collect detailed measures of hearing and vision which may be helpful in better understanding participant difficulty with using technology. As noted in the results, five participants showed changes in aphasia classification per the WAB-R Classification Criteria between first and second administrations. While the actual scores changed minimally for these individuals, the scoring range for the classification subtypes appear to be altered by these minimal changes. This variability is consistent with previous work 16 demonstrating that clinical impressions may differ from assessment classification. For example, one group found only 64.3% agreement between WAB-R classifications and bedside classification by clinicians for individuals with acute aphasia (John et al., 2017). This discrepancy indicates the need to rely on not just standardized test classification, but also clinical judgment in assessing aphasia subtype. While three participants showed greater than 5-point differences in the two test administrations, no clear pattern of change emerged in terms of domain scores, the modality of administration or the order of administration. Importantly, given the lack of trend in 5-point differences and aphasia classification scores related to in-person vs. videoconference assessment and first vs. second administration, these differences appear to be due to individual variability and not a.) systemic differences between videoconference and in-person assessment or b.) certain participants receiving treatment during the course of the study. Further, Villard and Kiran observed that stroke-induced aphasia may co-occur with difficulty maintaining consistent performance for attention tasks; this may have implications for both linguistic and non-linguistic task performance (Villard & Kiran, 2015). Of note, one of these cases appeared to be due to a slight variation in administration between evaluators, highlighting the importance of test administration training and inter-rater calibration. In conclusion, we believe there is a need for more validation of videoconference assessment administration in this population. Further work should expand to larger sample sizes, a more diverse patient population and a variety of assessments for individuals with aphasia, including traditional standardized assessments as well as natural measures of conversation. It is important to understand the kinds of modifications for videoconferencing that can be made to aphasia assessments while maintaining performance equivalent to in-person test administration. 17 Future study in this field should also include more detailed information regarding hearing and vision that could affect participation. In addition, future work should determine how best to engage caregivers in tele-assessment, including furthering our understanding of barriers to troubleshooting technological difficulties in the home environment. 18