Between September 2022 and April 2023, a total of 27 people with DCM were enrolled in the prospective and decentralized EMPOWER study [22]. Prospective participants were recruited via a web-based campaign and asked to complete consent and registration forms (Figure 1) [23,24]. These were used to screen participants for eligibility. Participants were deemed eligible if they had a self-reported diagnosis of DCM, owned a smartphone, and were able to stand and walk without the assistance of another person. Eligible participants were invited to download the MoveMed app to their smartphones and complete an electronic, baseline questionnaire on neuromuscular function, hand dominance, and quality of life. This included questions from the patient-derived mJOA (P-mJOA) and the World Health Organization Quality of Life Brief Version (WHOQOL-Bref).

All enrolled participants were asked to perform each task in the MoveMed app once per week for a period of 12 weeks. Task adherence was remotely monitored once a week using a bespoke web-based dashboard. Participants were offered reminders and help via email once a week if 14 days passed since the completion of the latest task. These were offered a total of 2 consecutive times per participant, after which the participant was considered lost to follow-up. At weeks 6 and 12, participants were asked to complete the same electronic questionnaire from week 1.

MoveMed is a smartphone app designed by academic neurosurgeons and computer scientists from the University of Cambridge to administer PerfOMs (Figure 2). These may be administered by clinicians during in-person visits or self-performed by individuals in the community. Version 1.0.0 of the app originally offered 3 performance tasks: a fast tap test, a hold test, and a stand and walk test. Version 1.2.2 incorporated an additional offering—a typing test—while making no changes to the 3 original tasks. Versions 1.0.0 and 1.2.2 were available in the Android Google Play Store and iOS App Store, respectively, at the time of writing and were used in this study by enrolled participants.

The fast tap test is a unidimensional PerfO task that assesses finger dexterity through a 6-second smartphone touch-based task. Users are shown a demonstrative cartoon (Figure 2A) and instructed to “touch the center of the target with [each] hand as many times as possible.” In-app video demonstration is also available. The construct (finger dexterity) is assessed by measuring the speed, accuracy, and efficiency of finger tapping as continuous variables and analyzing them as a panel of unidimensional measures. Content validity was assessed by AYT, MRNK, and BMD through literature review and clinical and patient input and deemed relevant, comprehensive, and comprehensible at the time of development [25-27]. In this study, tap latency was used as a reflective measure of finger dexterity.

The typing test is another unidimensional PerfO task that assesses finger dexterity through a 2-stage smartphone touch-based task. Users are shown a demonstrative cartoon (Figure 2B) and instructed to “type as correctly as they can, without rushing.” In-app video demonstration is also available. The construct (finger dexterity) is assessed by measuring the speed, accuracy, and efficiency of typing as continuous variables and analyzing them as a panel of unidimensional measures. Content validity was assessed by AYT, MRNK, and BMD through literature review and clinical and patient input and deemed relevant, comprehensive, and comprehensible at the time of development [25-27]. In this study, typing speed was used as a reflective measure of finger dexterity.

The hold test is a unidimensional PerfO task that assesses upper limb stability through an 8-second in-hand smartphone task. Users are shown a demonstrative cartoon (Figure 2C) and instructed to “hold the phone, screen up in the palm of [their] outstretched hand.” In-app video demonstration is also available. The construct (upper limb stability) is assessed by measuring the involuntariness, rhythmicity, and oscillation of the upper limbs as continuous variables and analyzing them as a multidimensional Stability Score. Content validity was assessed by AYT, MRNK, and BMD through literature review and clinical and patient input and deemed relevant, comprehensive, and comprehensible at the time of development [25-27]. In this study, the Stability Score was used as a reflective measure of upper limb stability.

The stand and walk test is a multidimensional PerfO task that assesses gait through a 2-stage in-hand smartphone task. During the first stage, users are instructed to “sit upright on the edge of a chair [and to] press the green button [when they are ready to] stand and remain still.” During the second stage, users are instructed to “walk [in] any direction.” In-app cartoons and video demonstrations are also available (Figure 2D). The construct (gait) may then be assessed by measuring standing or walking as continuous variables and analyzing them as multidimensional measures. Content validity was assessed by AYT, MRNK, and BMD through literature review and clinical and patient input and deemed relevant, comprehensive, and comprehensible at the time of development [25-27]. In this study, cadence was used as a reflective measure of gait.

The 2 patient-reported outcomes (PROs) or PRO measures (PROMs) were used as comparators for DCM: the P-mJOA and the WHOQOL-Bref.

The P-mJOA score is a multidimensional, patient-reported questionnaire that assesses neuromuscular function in DCM across 4 items: motor dysfunction of the upper extremities (MDUE), motor dysfunction of the lower extremities (MDLE), sensory function of the upper extremities, and sphincter dysfunction [28]. Responses are scored on an ordinal scale per item and presented as both a panel of unidimensional scores and an unweighted sum-total, multidimensional score. The P-mJOA score was selected due to the existence of a systematic assessment of construct validity (r>0.5) and feasibility in DCM [6] and due to the use of its clinically reported analog (the mJOA) as the current gold standard. The P-mJOA score was favored over the mJOA score since it is intended to be a truly patient-reported equivalent of the mJOA score, which can be understood by individuals with no medical knowledge or training [29].

The WHOQOL-Bref is a multidimensional, patient-reported questionnaire that assesses quality of life across 26 items grouped into 4 domains: physical health, psychological health, social relationships, and environmental health [30]. Responses are scored on a 5-point ordinal scale per item and presented as a panel of sum-total, multidimensional scores. Responses to 2 items may, furthermore, be presented individually to give insight into the respondent’s global perception of their quality of life and their quality of health. These were presented in writing to describe the population’s characteristics but were not considered robust enough to warrant correlation analysis. The WHOQOL-Bref was selected due to the existence of systematic assessments of validity, reliability, and responsiveness in traumatic brain injury [31], Parkinson disease [32], and DCM [6]. It was also favored over the 36-Item Short Form Health Survey due to its relative brevity and over the EuroQOL Five Dimensions Questionnaire due to licensing restrictions.

The COSMIN manual defines validity as “the degree to which [an instrument] measures the construct it purports to measure” [33]. In the absence of a gold standard, validity may be assessed formally through hypotheses testing of correlations to known standards. These may then be judged both as a panel of stand-alone ratings [34].

In this study, we assessed validity by correlating the MoveMed PerfOs to their corresponding patient-reported comparators. This was achieved by comparing it to the P-mJOA and WHOQOL-Bref PROMs. Due to the ongoing status of the trial, data from the first 3 weeks of the study were included (Figure 1). All available tests within this period were included. Longitudinal replicates of MoveMed tasks were averaged before comparing their mean scores to the mean scores of the PROMs. Responses from the baseline questionnaire were used and results were subgrouped by diagnosis. Missing data were not imputed, and all analyses were done using Python (version 3.10.12; Python Software Foundation).

The goal of the analysis was to determine “whether the direction and magnitude of a correlation is similar to what could be expected based on the constructs that are being measured” [33,35]. Spearman rank correlation coefficients (ρ) were thus computed due to their suitability for ordinal scales. In accordance with COSMIN, P values were not used “because it is not relevant to examine whether correlations statistically differ from zero” [33,35]. Hypotheses about the direction and magnitude of correlations were instead drawn and adapted from COSMIN [33] and de Vet et al [36]. We hypothesized that the magnitude of correlations between outcomes measuring similar constructs should be ≥0.5; the magnitude of correlations between outcomes measuring related, but dissimilar, constructs should be ≥0.3, and ideally <0.5; the magnitude of correlations between outcomes measuring unrelated constructs should be <0.3; and the direction of correlations between outcomes measuring directly related constructs should be positive (>0) and negative (<0) between outcomes measuring inversely related constructs.

As reported in Yanez Touzet et al [6], constructs were defined as “similar” if they both measured the same domain with a unidimensional instrument. If they measured the same domain, but at least 1 of the instruments was multidimensional, the constructs were defined as “related but dissimilar.” Constructs measuring different domains were otherwise defined as “unrelated.”

The COSMIN Risk of Bias checklist 9a [33] was used to assess the methodological quality of hypotheses testing.

Overall assessments of construct validity were made using a panel of ratings and prior knowledge of content validity. These were appraised qualitatively and presented in writing due to the relatively higher importance of some comparators over others. As in COSMIN [33], correlations were converted into ratings by comparing results to hypotheses. Correlations in accordance with hypotheses were rated “sufficient.” Correlations in opposition were rated “insufficient.” Correlations in between boundaries (eg, ρ=0) and statistical artifacts (eg, nonmonotonic data) were rated “indeterminate.”

This study was independently assessed and approved by the University of Cambridge (HBREC.2022.13). All study participants provided informed consent before enrolling in the study and were able to opt out at any point. Study data were anonymized. None of the participants received any form of compensation for enrolling in or completing the trial.

A total of 27 participants with DCM enrolled in the prospective and decentralized EMPOWER study (Figure 3), principally via advertisement through Myelopathy.org, a DCM charity [23,24]. On average, participants were aged 60 (SD 11) years (Table 1). DCM severity ranged from mild to severe (P-mJOA total score range 8-18). The impact on upper limb motor function ranged from none to “unable to eat with spoon but able to move hands” (P-mJOA MDUE subscore range 2-5) and the impact on lower limb motor function ranged from none to “able to move legs but unable to walk” (P-mJOA MDLE subscore range 2-7). Overall health perception ranged from “satisfied” to “very dissatisfied” (WHOQOL overall health range 1-4), and overall quality of life perception ranged from “very good” to “very poor” (WHOQOL overall quality of life range 1-5). In terms of the MoveMed PerfOs, participants paused for 80-2600 ms in between taps and typed approximately 0.6-2.5 keys per second. Arm stability ranged from 39% to 100% and cadence ranged from 14 to 112 steps per minute.

Differential app use was noted throughout the studied period (Table 2). More participants used the stand and walk and typing tests (n≥20) than the fast tap and hold tests (n≥12). However, mean adherence was higher with the fast tap and hold tests (100% and 90%, respectively) than with the stand and walk and typing tests (77% and 72%, respectively). Crucially, median adherence was satisfactory: 100% for the fast tap, hold, and stand and walk tests, and 80% for the typing test. Differential use was thus attributed to individual test preferences.

Spearman rank correlation coefficients are reported in Table 2. As expected, correlations were positive between PerfOs and PROs measuring directly related constructs (eg, hold and typing tests vs P-mJOA and WHOQOL-Bref) and negative between PerfOs and PROs measuring inversely related constructs (eg, fast tap test vs P-mJOA and WHOQOL-Bref). This was most pronounced in the fast tap, hold, and typing tests.

Correlation magnitudes were, furthermore, highest between PerfOMs and PROMs of neuromuscular function (eg, fast tap test vs P-mJOA≥0.3) and lowest between PerfOMs and PROMs of quality of life (eg, fast tap test vs WHOQOL-Bref<0.3).

This was also in accordance with expectation and was most pronounced in the fast tap, hold, and typing tests.

Correlation magnitudes were notably low (<0.3) in the stand and walk test. This could be due to it being the only multidimensional PerfOM in the battery. Importantly, correlation with the lower limb comparator domain (ie, the P-mJOA MDLE subscore) was the highest, in accordance with expectation.

No risk of bias factors from the COSMIN Risk of Bias checklist were recorded (Multimedia Appendix 1). This was equivalent to a “very good” rating for methodological quality [33].

Hypotheses and result ratings are also reported in Table 2. These are appraised in writing due to the relatively higher weight of some comparators over others.

Overall, 74% (67/90) of correlations for the fast tap, hold, and typing tests were in correspondence with hypotheses (Table 2). This provides robust evidence for the validity of these PerfOMs in the assessment of DCM: particularly due to the relatively higher importance of the correlations to the upper limb comparator (ie, the P-mJOA MDUE subscore), which were concordant. For the stand and walk test, 47% (8/17) of the correlations were in correspondence with the hypotheses. This also provides preliminary evidence for the validity of this PerfOM in the assessment of DCM: particularly due to the relatively higher importance of the correlation to the lower limb comparator (ie, the P-mJOA MDLE subscore), which was concordant. Taken together, these data provide “very good” quality evidence for the overall validity of the PerfOMs in the assessment of DCM.

Smartphone apps are increasingly being used to administer clinical outcome measures in medicine. This study used consensus-based standards to assess the validity of an app designed by neurosurgeons and computer scientists from the University of Cambridge. A total of 2 lines of evidence were produced: first, a panel of correlations between the app’s tasks and established clinical comparators, and second, a panel of ratings made in accordance with prespecified hypotheses. The former produced modular evidence of construct validity and the latter a means for its overall appraisal. This type of evidence corresponds to clinical validation under the V3 framework for biometric monitoring technologies and succeeds in laboratory-based verification and analytical validation [21].

Construct validity uses comparison to other measures to assess validity. Where comparators take different approaches or contain their own limitations, validity should not be defined by traditional correlation thresholds [37]. This is applicable to DCM, where we are trying to improve disease measurement. For example, we recognize the mJOA score as a gold standard measure of disease severity, but it measures multiple constructs with limited discrimination, particularly of milder diseases. If a new measure has a correlation of 1.0 with an existing measure, it indicates that the 2 instruments are equivalent, which suggests it is unlikely to offer any improvement. For assessing construct validity, it is therefore preferable to explore expected relationships through hypothesis testing. As expected, the direction and magnitude of MoveMed correlations were most convergent between tasks and questionnaires measuring similar constructs than tasks and questionnaires measuring dissimilar constructs (eg, fast tap test vs P-mJOA > fast tap test vs WHOQOL-Bref). This is because neuromuscular tasks should correlate more with neuromuscular constructs than with non-neuromuscular ones (eg, finger dexterity vs upper extremity neuromuscular function > finger dexterity vs quality of life) and because unidimensional tasks should correlate more with other unidimensional measures than with multidimensional ones (eg, unidimensional vs unidimensional>unidimensional vs multidimensional).

To enable performance across correlations to be judged, a proportion of overall hypothesis agreement may be used [34]. After rating, 74% (67/90) and 47% (8/17) of unidimensional and multidimensional results, respectively, were deemed sufficient for construct validity in the DCM subgroup. In the absence of risk of bias factors, these data provide “very good” quality evidence for the validity of MoveMed tasks in DCM.

The standards adopted by this study have been previously used in the assessment of PerfOMs by authors of the COSMIN guidance [33]. While not originally designed for this purpose, these standards are considered to be a cornerstone in clinimetric validation and, importantly, overlap with industry guidance from the US Food and Drug Administration [17,38]. This study thus made a point to conduct and report the COSMIN Risk of Bias assessment to aid the reader in their interpretation of the rating panels (Table 2).

While construct validity testing (often criterion validity) is more commonly used by investigators, correlation coefficients require interpretation, as outlined. For similar reasons, the relative performance of instruments should not be judged solely based on the magnitude of correlation coefficients. This is reflected in clinimetric standards which instead recognize content validity as the most important arbitrator of validity and wider performance. Content validity uses stakeholder judgment and feedback to determine validity and will be further reported separately for MoveMed, following study completion. When developing and reviewing measurement instruments, understanding clinimetrics is therefore critical.

In this cohort, the impact that DCM would be expected to have on the P-mJOA and WHOQOL-Bref was similarly seen on the fast tap, hold, typing, and stand and walk MoveMed Tests. Correlations with total scores and limb-specific subscores were recorded, in accordance with prespecified expectations. The most interesting finding was the strong correlation between the P-mJOA MDUE subscore and the hold test Stability Score. This is because upper limb stability is not classically thought to be a marker of DCM. The authors attribute this finding to the composite nature of the upper limb stability construct, which includes elements of arm strength, muscle fatigue, and balance. Further studies will follow-up with more data on the subject (eg, content validity). This may very well be an example of a subclinical phenomenon that the human eye cannot catch but that mobile sensors can.

An important strength of this study is its design by individuals with formal training in clinimetrics. This is reflected in the absence of risk-of-bias factors from the COSMIN checklist in Table 2 and the study’s reporting. There is, unfortunately, a general paucity of well-designed clinimetric studies in the literature [33,34,39-41]. The use of the COSMIN manual is thus strongly encouraged by the authors. Another strength of this study was the use of PerfOMs that can collect several measurements quickly, ecologically, and longitudinally. This means that the construct should be captured more precisely, more reflective of pathology in the patient’s natural environment, and potentially more responsive to intervention. In the future, these hypotheses will be formally assessed via further clinimetric studies.

Despite its conscientious design, this study has limitations. First, standards for patient-reported methods were adapted to assess performance-based methods. This was done to overcome the absence of standardized criteria in this field and because there is precedent for it in Terwee et al [34] and the COSMIN manual [33]. Second, this study reports on 27 individuals (7 months of recruitment). The COSMIN standards are known for being rigorous (or stringent) and, ideally, at least 50 participants should be included to earn a modified Grading Of Recommendations, Assessment, Development, and Evaluations (GRADE) score of “high” [33,34,39-41]. Third, we assumed that the constructs of all WHOQOL-Bref domains would be dissimilar to the PerfOMs but this may not be the case. The WHOQOL-Bref, ultimately, contains questions on physical activity, and the relatedness of this construct to the fast tap tests and the typing tests may have been observed in Table 2. Fourth, people with a severe form of the disease may have been excluded from enrollment. This would be due to the exclusion of individuals who were unable to stand and walk without the assistance of another person. The potential risks of remote participation in this subset of individuals, however, were deemed to outweigh the benefits by the ethical committee. Further in-person research could address this limitation in the future.

This study provides initial evidence for the validity of the MoveMed PerfOMs in the context of adults with DCM in the community.