Ethics approval was obtained from the Ottawa Health Sciences Network Research Ethics Board (20230311‐01H) in June 2023, followed by the Bruyère Research Ethics Board (M16-22-006) and the University of Ottawa Board of Ethics (H-06-23-9348) in June and July 2023, respectively. Patient-participants verbally consented over the telephone or provided informed consent in person. Privacy and confidentiality were maintained. Participants were provided with a parking voucher and CAD $30 (US $22) gift card following completion of participation in the study.

Each patient-participant completed both the in-person and remote assessment at TOHRC on the same day. The assessments were conducted during a scheduled in-person appointment for outpatients, or a specific appointment set up for inpatients. The 2 assessments were separated by a brief rest period [32] with additional rest time provided until the participant could complete the remainder of the measures.

The order of the assessments (in-person and remote) was randomized and counter-balanced to ensure an equal number of participants completed the in-person and remote assessments first. Randomization occurred through REDCap (Research Electronic Data Capture; Vanderbilt University) using a random numbers table. The rationale was to equalize the influence of fatigue and learning effects on the subsequent assessment. The assessments consisted of the same clinical measures (Table 1).

For the remote assessment, the patient-participant was in a separate room from the assessor. All patient-participants used the same computer and received technical support from a research team member if needed. Safety precautions during the remote assessment included (1) the presence of a research team member in the room throughout the remote assessments and (2) positioning patient-participants in front of a wall, chair, or bed during balance testing. All remote assessments used a Dell Vostro 3520 Laptop (with a 15.6-inch display) running the Microsoft Teams platform (for the patient-participants). Assessors used desktop computers. The remote assessments were audio-video recorded for later review and evaluation by a different clinician. For the in-person assessment, a research team member was present in the room to record the assessment using the same laptop as the remote assessment. Safety precautions were provided by the assessing clinician.

Two assessors (rater A and rater B) documented clinical findings for each patient-participant. Rater A completed the in-person and synchronous remote assessments. Rater B documented findings asynchronously from audio-video recordings of the remote assessment at 2 time points separated by approximately 1 month. Figure 1 outlines the study assessment procedures.

Feedback was obtained from both clinician- and patient-participants using a feedback form, following completion of the assessments. Participants answered questions related to the environmental setup of the remote assessment, confidence in the assessment findings, perceived similarity between the 2 approaches, and provided any additional feedback on the form.

A total of 20 patient-participants completed both the in-person and remote assessments (see Table 2). In it, 15 (75%) participants were female. Most participants were working at the time of their assessment. Most participants reported limitations in functional abilities and perceived their mental health as fair to poor. When considering the criteria outlined regarding the identification of abnormality on each measure, 18 out of the 20 participants were abnormal on at least 1 of the measures in the in-person assessment.

The injury characteristics of the 20 patient-participants are presented in Table 3. A total of 14 (70%) participants sustained a concussion, and the remainder sustained a moderate to severe traumatic brain injury or other form of ABI, with most injuries occurring outside of the workplace context.

Most participants used technology daily and the majority rarely needed assistance when using technology (Table 4).

Three clinicians participated as assessors. Two assessors (male and female), acting as rater A, were a physiatrist and a physician assistant. The third clinician (a male), acting as rater B, was a physiatrist and observed the recordings of the assessments. These clinicians typically assess more than 50 patients with ABI annually and reported feeling confident in their ability to complete the in-person and remote assessments.

We experienced challenges recruiting clinicians willing to conduct the study assessments, and clinicians’ scheduling conflicts posed difficulties with patient-participant recruitment; therefore, the involvement of additional professionals, such as physiotherapists, will be needed to support the large-scale study. We recruited, on average, 1 patient-participant per week at TOHRC. A total of 38 potential participants were identified. In it, 7 could not be reached. Of the potential participants reached, 6 (23%) declined, primarily due to concerns that the multiple assessments would make their symptoms worse. Our rate of recruitment was, therefore, 20/33 (61%) of people approached and 20/26 (77%) of people reached. Given the recruitment rate, at 1 center, we anticipate being able to approach approximately 6 participants per month. With the anticipated recruitment rate of 61% over 5 months, we can feasibly recruit 20 participants, which would mean the future large study with a target sample size of 60 [41] would require 15 months to complete recruitment.

The time required to complete the in-person assessment ranged from 5 to 13 minutes and averaged 9 minutes. The time required to complete the remote assessment ranged from 7 to 26 minutes and averaged 13 minutes.

The assessors expressed high confidence in their findings on the in-person (20/20, 100%) and remote (19/20, 95%) assessments. Only on 1 occasion was an assessor “neutral” in terms of their confidence in their findings on the remote assessment.

Most patient-participants agreed or strongly agreed that they felt confident in their assessors’ findings on the in-person (19/20, 95%) and remote (15/20, 75%) assessments. Two patient-participants did not feel confident and 2 were “neutral” in their confidence levels on the remote assessment. One participant did not feel confident in both the in-person and remote assessments.

The preliminary sensitivity of the remote compared to in-person administration of the measures ranged from 0.0 to 1.0 (Table 5). This suggests poor (finger-to-nose testing) to excellent (near point convergence) ability to detect deficits in the remote assessment when deficits are truly present (based on the reference standard, which is the in-person assessment). On 2 occasions, evaluation of saccades was not documented by the assessor in error. On 1 occasion, the VOMS was not completed due to the patient’s inability to complete the measure because of aggravation of symptoms. No abnormality was documented for effort on both the in-person and remote assessment and, therefore, sensitivity could not be computed.

Table 6 presents the preliminary information on the interrater and intrarater reliability of the measures when administered remotely. Cohen κ values for interrater reliability ranged from poor for balance testing (0.38), range of motion (0.47), and saccades (0.49) to excellent (1.0) for VOMS change in symptoms. The intrarater reliability ranged from poor (0.44) for the evaluation of saccades to very good (0.89) for VOMS change in symptoms.

Most (17/20, 85%) participants reported an increase in preexisting brain injury symptoms, such as increased headaches, dizziness, and nausea, during the VOMS measure; however, this was true for both the in-person and remote assessments.

This is one of the first studies to examine the preliminary psychometric property information of concussion assessment using remote approaches. According to Montes et al [42], a systematic approach to remote assessment development is essential, with established in-person measures serving as a basis for comparison or as a reference standard. Taking this into consideration, we first carried out reviews of the psychometric properties of potential measures to help select the measures to be included in our remote assessment [14,22,23]. In this study, we report on the acceptance, feasibility, and preliminary psychometric properties of these assessments to provide initial evidence of the remote assessment and inform a larger study of remote assessment for concussions.

The rate of recruitment is a critical indicator of the success of a future large-scale study. Our moderate rate of recruitment indicates that there is sufficient interest among people with ABI to participate in a remote assessment-based study; however, additional strategies may be needed in order to increase our ability to reach potential participants. Although patient-participants showed a moderate willingness to participate, we did experience more difficulties recruiting males and recruiting people with other forms of ABI when compared to the recruitment of people with concussions. This was mainly because the practice of one of our assessors only included people with concussions (and not other forms of ABI). We experienced some difficulty recruiting people with known abnormality on specific tests, particularly those with abnormal coordination, as people with concussion typically have normal coordination and recruitment of people with concussion was easier due to the nature of the practice of our recruited assessors. The rate of patient recruitment appeared to be influenced by clinical status (concerns regarding aggravation of symptoms due to the nature of the measures and the need to complete the measures twice). Hunt et al [43] reported that stakeholder engagement in concussion research may be inhibited by injury-related factors, personal deterrents (vulnerability and fear), and environmental barriers. Concussion symptoms, including physical, cognitive, and emotional, were identified as a barrier to involvement by patient-participants [43].

The length of time required to complete our study procedures is another essential aspect of evaluating the feasibility of completing a large-scale study and participant burden. Factors influencing the time required to complete the study assessments included the participant’s ability to follow instructions, participant symptom aggravation, and technology issues (internet speed). The average time required to complete our remote assessment was longer than the in-person assessment, which is contrary to findings reported by Tran et al [44] who noted that remote visits tend to be similar in length when compared to in-person visits. However, adaptation of certain measures to remote environments, such as the VOMS, has been previously reported to take time and practice [45], which may have contributed to the additional time required to complete the remote assessment. While the duration of the remote assessment was longer than the in-person one, the reduced travel time required to attend an in-person visit highlights the convenience associated with remote assessments from the patient-perspective [46,47], although increased clinician time required may be a concern [14].

Additionally, having technical support with camera angles may have positively impacted the experience of the participants with the remote assessment in this feasibility study. A research team member was present to aid with moving the laptop to improve the visibility of the patient for the clinician, close blinds, and troubleshoot internet issues. Ownsworth et al [48] reported that ongoing access to support could improve user-friendliness and facilitate the use of remote care. Further, there is a need for reliable and high-quality videoconferencing technology, which could present a challenge in practice due to variable accessibility and cost [49]. To improve the issue associated with lighting, which was expressed by participants, it is recommended that blinds are closed when completing remote assessments and lights are bright enough for clinicians to observe the individual on screen but are manageable for the individual in terms of limiting symptoms aggravation. To improve visibility, a blank backdrop is recommended along with appropriate positioning of the camera at eye level [50].

Participants perceived the environmental setup of the remote assessment to be adequate. The feedback obtained highlighted the advantages of remote assessments over traditional in-person assessments, which is consistent with the literature, including eliminating the need to drive to the assessment center, and the ability to control the environment better at home, such as having the capacity to dim lights [46,51,52]. However, home can present challenges as well [53]. The remote assessments for this study were all conducted using the same laptop and within the same setting, which may have positively impacted the experiences of the participants. The home environment may present unique challenges, such as distractions and variable screen sizes, which in turn may impact outcomes on the assessment [54]. It is recommended to develop a plan and schedule for these assessments to minimize such distractions in the home setting [54]. Further, 1 participant in our study was homeless and was admitted as an inpatient, and therefore, would not have access to needed technology in a home environment.

When remote assessments are implemented in practice, it is important to ensure that findings obtained through the remote assessment are comparable to those obtained through in-person approaches [27]. For the most part, the clinician- and patient-participants in this feasibility study perceived that though findings were similar in the in-person and remote assessment. This perception of congruence is supported by objective data obtained by Vargas et al [55] who examined the feasibility of remote assessment of concussion using a telemedicine robot in which a neurologist remotely assessed injured athletes simultaneously to sideline provider in-person assessments. Vargas et al [55] provide preliminary information on the strong level of agreement (within 3 seconds or points 100% of the time) between findings documented by the in-person sideline provider and the remote neurologist on specific concussion measures (Standardized Assessment of Concussion [a cognitive test], modified Balance Error Scoring System [a balance test], and King-Devick test [a saccadic eye movement test]) [55]. While the perception of congruence was high, participants, both assessors and patients, were more often more confident in the findings of the in-person assessment when compared to the remote assessment. These findings are in line with Gilbert et al [56], who noted that clinicians and patients were satisfied with remote consultations; however, in-person consultations are still preferred (outside the COVID-19 pandemic).

In-person measures should have acceptable reliability properties for method-comparison studies to have meaningful results [27]. It is, therefore, recommended to determine the reliability properties of the remote assessment as part of the method-comparison study. Sensitivity metrics of the measures included in the remote assessment are also of interest, as clinical assessment findings are relied upon to detect deficits and guide intervention [2]. The preliminary investigation of reliability and sensitivity in this study appears to vary when compared to previously reported in-person values. Reliability values for in-person administration of the measures range from moderate (test-retest reliability of the single leg stance, with a κ of 0.43 [57]), to excellent (within-tester reliability of cervical spine range of motion with an ICC of 0.90 [58]). The sensitivity metrics associated with in-person administration of the measures range from moderate (0.45 for balance testing in people with traumatic brain injury [59]) to excellent (96% for the VOMS assessed in people with concussion [58]). The findings of this feasibility study indicate that the evaluation of finger-to-nose testing, saccades, and cervical spine range of motion appear to have poorer properties associated with remote administration when compared to in-person metrics, which have documented sensitivities of 71% [59], 64%‐77% [60], and 86%‐95% [61], respectively.

Consistent with the literature, the lack of physical presence in the remote assessment may have been associated with an inability to gain a full understanding of the patient’s status in these measures and subtle abnormalities such as those observed during oculomotor assessments may have been more difficult to capture through videoconferencing [62,63], which was also subjectively acknowledged by patient-participants in this feasibility study. Documentation of psychometric properties of measures administered in both in-person and remote contexts is required to support the hybrid approach to care, which integrates both in-person and remote interactions with clinicians. This approach is desired by both clinicians and people living with concussions [13,14].

The data obtained in this study suggest that there are specific measures, such as the evaluation of saccades, cervical spine range of motion, and finger-to-nose testing that are more difficult to administer remotely and, therefore, a high level of care in delivery should be considered to increase reliability and sensitivity. Further development and research are, therefore, needed in this area to determine how best to administer these measures remotely. This could include using advanced technologies, improving training of clinicians, or improving administration instructions [50], all of which will be considered for the large-scale study. Best practices for administering these measures need to be identified, and strategies to overcome limitations posed by the remote environment should be explored.

A strength of this study includes the use of technology and software that are commonly used to complete remote assessments at TOHRC. This decision was made to mimic as best as possible the current clinical approach to remote assessments. Further, we expanded the inclusion criteria to include participants with various forms of ABI, which ensured the inclusion of individuals with identifiable positive findings. This approach not only facilitated recruitment but also enhanced the generalizability of our findings.

Our study also has several limitations. First, this study was carried out in a controlled environment to standardize as many aspects of the remote assessment as possible. Thus, we did not test the remote assessment in real-world settings such as with people present in remote or rural regions, where factors such as home environments, technology used, internet, and so forth, may have influenced reliability. Second, due to an inability to complete the assessments more than twice, we were unable to establish the reliability properties of the in-person assessment and relied on the literature for the in-person properties needed for comparisons, which will also be required for the large-scale study. Third, it was not feasible for 2 different clinicians to conduct the initial in-person and remote assessments in our clinical environment due to scheduling constraints, so the same clinician completed both initial study assessments. In an attempt to mitigate potential bias and in line with best practices for method-comparison studies comparing in-person and remote approaches [27], we randomized the order of the study assessments and compared the in-person assessment conducted by rater A with the findings of the observed recording completed by rater B; however, this approach is limited in that rater differences may have impacted the findings. Finally, the widths of the CIs reported in this feasibility study are extremely wide (due to the small sample size) and, therefore, there is little conclusion one can draw from the psychometric property estimates. The small sample size further limits the generalizability of the study findings. More data and extensive study are needed to definitively establish the reliability and sensitivity properties associated with the remote concussion assessment.

For clinical measures to be confidently used by clinicians in remote care practice, comparisons need to be made to their in-person counterparts. Given the width of the CIs, little can be concluded regarding the sensitivity of the concussion measures administered remotely, when compared to in-person administration, and the reliability of those measures. However, this feasibility study documents the time needed to complete components of a concussion assessment remotely and confirms the probable safety of the assessment, with no adverse events specific to the remote assessment documented. The findings of this feasibility study provide a foundation and will inform processes for a planned follow-up study that contains an adequate sample size to estimate the psychometric properties with adequate precision. Future work should expand on this foundation through the exploration of the impacts of home environments on remote concussion assessment outcomes and through the investigation of additional relevant psychometric properties, such as responsiveness.