Digital and mobile health technologies, including smartphone-based monitoring and wearable devices, have a wide range of applications in clinical practice [1-3]. A clinical “use case” describes how a technology can be implemented in a clinical context, including the expected benefit and expected beneficiary. Clinical use cases are essential for determining the outcomes to be used in trials evaluating effectiveness as well as for obtaining regulatory approvals and explaining the benefits of a health care technology to potential funders and patients. The adoption and scaling of novel technologies in health care are dependent on a well-defined use case with a clearly defined problem to be addressed [4]. The inclusion of clinicians in the development of such technologies is known to be important for successful implementation, as it ensures the appropriateness of technology for the specific requirements of patients and the health care system [5].

Remote Assessment of Disease and Relapse–Central Nervous System (RADAR-CNS) was a 6-year project to understand the feasibility and acceptability of using remote measurement technology (RMT) to collect health-relevant data from individuals living with epilepsy, multiple sclerosis (MS), or depression [6]. The project was a collaboration across 6 European countries (Denmark, Germany, Italy, the Netherlands, Spain, and the United Kingdom) and has involved the development of a bespoke, open-source platform RADAR-base. The platform collates data from commercially available Fitbit smart watches measuring activity, heart rate, and heart rate variability; the Empatica E4 wrist-worn epilepsy seizure detection device; Bittium Faros accelerometer and electrocardiogram Holter devices; and bespoke apps for passive sensing and active collection of user-entered data (THINC-it) [7]. We refer to the combination of the platform, the apps, and the commercial devices as the RADAR-CNS RMT system. Observational studies have been conducted to establish the feasibility and acceptability of collecting data from individuals living with MS, epilepsy, or depression using these sensors, apps and platform to develop new predictive algorithms based on the data set [8-10]. Patient involvement has been conducted throughout the program, and patient focus groups and other involvement studies have been conducted in multiple European countries to elicit patient views and inform the RMT under development [10-13].

The aim of this study was to specify priority use cases for RMT in 3 central nervous system disorders (epilepsy, MS, and depression). An initial set of 13 use cases were developed through discussion with health care professionals (HCPs) and researchers working in each of the 3 clinical work packages within the project. The development of these use cases considered the fit to the target population, the potential for a positive impact on the health and safety of patients, whether the use case would offer an improvement on current methods, and the existence of prior evidence to support the use case. These were also informed by our prior work, which included: a small-scale survey with patient advisers, HCPs, and researchers [14]; in-depth interviews with HCPs [15]; and a large-scale survey of 1006 clinicians on the current and potential use of RMT and apps in clinical practice [16] and the potential value of remote measurement data [17]. The Delphi study then sought to prioritize among the 13 use cases (5 in epilepsy, 3 in MS, and 5 in depression) to determine which of these would be most practicable and useful in the eyes of the expert clinician panel, who were outside of the consortium and so offered a more objective point of view. The number of use cases included in the study was considered to be manageable without overburdening participants.

The use cases were also presented to the RADAR-CNS Patient Advisory Board (PAB) to seek further input ahead of this study in a short consultation via Microsoft Teams, with diagrams and descriptions of use cases provided by email in advance. The RADAR-CNS PAB includes members living with each of the 3 conditions from multiple countries across Europe. Illustrations of the 13 use cases are included in Multimedia Appendix 1.

The final use cases for epilepsy are as follows:

The final use cases for MS are as follows:

The final use cases for depression are as follows:

The aim of this work was to prioritize the use cases with the potential for the greatest benefit for further development according to the views of HCPs external to the project. We sought to establish the type of benefit that each might offer according to a medical device design framework [18]. We also aimed to explore further related issues:

We aimed to seek a consensus in these areas, where prior work has shown that there is a disagreement between HCPs. In addition, we aimed to determine which prior known concerns about the use of RMT in clinical practice would actually prevent or discourage HCPs from using RMT with patients. As we recruited an international sample, we were also interested in exploring how the potential implementation of RMT might differ between countries.

The Delphi methodology has been widely used in health care research to gather expert opinions [19-21]. Key characteristics of the Delphi method include consultation of experts, elements of iteration and feedback to participants to enable a form of communication between them, and statistical methods used to summarize group responses to ensure the robustness of analyses [22]. Delphi studies can be used without face-to-face contact while still enabling the gathering of group opinions, which is of benefit when those whose opinions are required have busy schedules (eg, in clinical settings) or may be located across multiple countries [23-25], as is the case in this study. There were also obvious benefits to this approach during the COVID-19 pandemic.

Delphi studies feature multiple survey rounds, with feedback given to the participants between each. For example, Murphy et al [21] used a 3-round model to gather views and opinions on the potential of digital tools for mental health in the United Kingdom. The first round of a Delphi study typically asks open-ended questions, and these are used to generate closed questions for a survey in the second round, often with Likert-style responses. In the third round (if there is one), participants review the summarized results from the prior round and are then able to change their responses if required [26]. This results in either greater consensus among the groups or sustained disagreement, both of which are of interest [25]. Other models may omit the first qualitative round [19] and may include follow-up interviews after the final survey round [27].

A total of 23 clinicians treating patients with epilepsy, MS, or depression were recruited, with participation from all 6 European countries where RADAR-CNS is active (the United Kingdom, Germany, Spain, Denmark, the Netherlands, and Italy) and with representation from clinicians treating each of the 3 clinical conditions (Table 1). We expected some dropouts between the first and second rounds but were able to retain 74% (17/23) of the round 1 participants in round 2. A total of 9 respondents completed the interviews.

The research team decided that a third survey round was not required: 97.4% (114/117) of question items had a high or very high level of stability of responses between rounds 1 and 2, indicating that a third round would have had limited benefit. Multimedia Appendix 5 [30] presents the results for consensus and stability for all questions in the survey.

The threshold for consensus was reached on 74% (26/35) of questions on the epilepsy use cases in the first round and 86% (30/35) of questions in the second round, demonstrating a move toward consensus. For all of these items, consensus was reached that respondents agreed or strongly agreed with the statements presented, rather than selecting “disagree,” “strongly disagree,” or “neutral.” Five of the questions moving to consensus in round 2 concerned the fifth epilepsy use case, using RMT for trend analysis, and indicated a change in views toward agreement that this use case would “enable” new possibilities, “facilitate” care (make care easier), “enhance” care, and benefit patients and clinical teams.

It is notable that in round 1, for epilepsy use cases 1 to 3, the question on “catalyzing” (speeding up) existing processes received fewer "agree" or above responses than all other statements, indicating less confidence that these use cases would speed up existing processes. The PAB identified this pattern, and their comments were fed back to the participants ahead of round 2. There was comparatively low stability for these questions between round 1 and round 2, suggesting that participants changed their minds about this question, perhaps in response to the PAB’s comments.

The point estimate for the Gwet agreement coefficient statistic fell in the “very good” strength of agreement range (0.80-1.00) for 94% (33/35) of items, indicating a very high overall stability of epilepsy responses between rounds 1 and 2.

The threshold for agreement was reached on 10% (2/21) of questions on the MS use cases in the first round and 0% (0/21) of questions in the second round, demonstrating a move away from consensus. Where consensus was reached, it was a consensus that respondents “agreed” or “strongly agreed” with the statements presented rather than selecting “disagree,” “strongly disagree,” or “neutral.”

Fewer participants completed the MS use case questions in the second round (n=3) compared with the number completing epilepsy questions (n=6) and depression (n=9). This meant that even when a majority of 67% (2/3) of participants expressed an opinion, this did not cross the threshold of 70%, requiring a unanimous vote for this to occur. This explains the comparatively lower number of questions showing consensus in the MS group.

The point estimate for the Gwet agreement coefficient fell in the “very good” strength of agreement range (0.80-1.00) for 81% (17/21) of items, indicating a high overall stability of responses between rounds 1 and 2.

In the depression use cases, the threshold for agreement was reached on 83% (29/35) of questions in the first round and 89% (31/35) of questions in the second round. Where consensus was reached, it was a consensus that respondents “agreed” or “strongly agreed” with the statements presented, rather than selecting “disagree,” “strongly disagree,” or “neutral.”

The point estimate for the Gwet agreement coefficient fell in the “very good” strength of agreement range (0.80-1.00) for all 100% (35/35) of items, indicating a very high overall stability of responses between rounds 1 and 2.

There were 19 questions on further considerations of RMT (clinical time, frequency of data collection, technical support requirements, usefulness or value of data, and payment and reimbursement), which were rated by all 17 participants who completed both rounds. Participants reached a consensus of “agree” or “strongly agree” on 42% (8/19) of questions in round 1 and maintained this level of consensus in round 2 (Multimedia Appendix 5). A smaller proportion of questions in this section reached consensus than those in the section on use cases. For 11% (2/19) of questions in this section, participants moved from no consensus to a consensus that they “disagreed” or “strongly disagreed” with the statement. These questions were that “receiving data on a patient’s condition would be an added burden” and that “mood scores need to be collected from patients at risk of mental health conditions on a daily basis.” A total of 9 question items in this range did not reach agreement in the first or second rounds. The Gwet agreement coefficient showed “very good” stability between rounds (Altman benchmark 0.80-1.00) on only 63% (12/19) of questions in this range, indicating greater changeability between rounds for these questions compared with those relating to the use cases.

There were 7 questions on concerns about RMT, which could be rated as a serious concern (“Would prevent use entirely”), a medium concern (“Would prevent use in some situations”), or a lesser concern (“Would not prevent me using RMT with my patients”). There was no consensus for any question in this set in round 2.

The stability of responses between rounds 1 and 2 for these questions was lower than that for other parts of the survey. The change in responses was not uniform in one direction or the other, and neither was there a distinct movement toward or away from extreme responses (rating a concern as severe or lesser), indicating less certainty in relation to these questions compared with other sections of the survey.

The purpose of this mixed methods, sequential explanatory Delphi study was to prioritize among use cases for RMT in central nervous system disorders, which had been cocreated with clinicians and patients within the RADAR-CNS consortium. The results from the study have identified those likely to be of the most practical use and clinical benefit. The study has also contributed knowledge on country- and context-specific factors affecting implementation and revealed areas of consensus and disagreement among HCPs on practical aspects of RMT implementation.

We would caution against overinterpretation of the consensus scores for MS, where only 3 participants responded in the second round. Unfortunately, we could not recruit more experts in MS during the time available for the study, despite extending the recruitment window and using multiple recruitment methods. This is a shortcoming of many studies seeking the views of MS clinicians, as there are few medics specializing in this condition. However, the combination of item ratings and interview findings relating to MS provides useful insights into how RMT could be used for patient benefits in this condition.

In addition, the aim of this study was to explore the applicability of the RADAR-CNS RMT system in 3 central nervous system disorders, limiting its relevance to other conditions or monitoring platforms. However, the methodology set out here will likely be of interest to others seeking approaches to evaluate the application of novel systems in health care, and the findings will be of interest to those developing a variety of digital interventions for the specific conditions discussed.

Our work highlights the potential value of the implementation of RMT for 3 central nervous system disorders, including which applications of RMT would enable something new, enhance existing care, speed up existing processes, or facilitate or make easier the care of patients [14,18]. These results indicate that clinicians would consider RMT patient data to have sufficient value that it would be worth a financial outlay to implement RMT in clinical practice. However, health economic evaluation is required to determine the cost-effectiveness of applying RMT in each of these conditions [48], and the choice to implement is likely to be determined by whether cost-effectiveness is judged to meet local criteria that can vary by country [49].

The findings from this study and our prior work provide an indication of where costs may be incurred if RMT is implemented in a health care service [17]. The costs are likely to include introducing staff roles to manage patient data and provide technical support. Technical support staff could assist patients in setting up and maintaining RMT devices and support clinicians in making the best use of patient RMT data. The extra time for clinical staff to review patient data would be cost incurring, where fewer patients can be seen, although if this results in improvements in care and thereby patient condition, there may be an overall improvement in efficiency. The devices themselves and their maintenance also incur a cost for a health care service wishing to implement them. Although many of the technical devices incorporated within the RADAR-CNS system are consumer-grade technologies that may be owned by patients, our findings suggest that there is a need to provide each patient with devices meeting specific standards of accuracy, adding to costs.

Participants largely suggested that decisions about the implementation of new technologies were top-down and that commissioners and health service leaders would need to be convinced of the benefits of RMT for it to be implemented. In the United Kingdom, commissioners are often involved in redesigning services to incorporate new and beneficial technologies or products [50]. The exception to this was Germany, where doctors who run their own services at a local level are able to work with their budget holders to decide on the implementation of particular technologies. However, it is expected that these technologies should be demonstrably both clinically effective and cost-effective. In the United Kingdom, the National Institute of Health and Care Excellence provides guidance on the evidence required for approval of digital technologies [51]. Similarly, in the European Union, the European Medicines Agency works with groups to develop novel health care technologies to provide scientific advice [52]. Close working with these organizations would facilitate the further development and evaluation of the RADAR-CNS RMT system.

RMT offers new possibilities for the assessment of epilepsy, MS, and depression by enabling new ways of caring for patients, enhancing existing processes, facilitating care, and, in some areas, catalyzing or speeding up existing processes. Our study shows promise for the use of wearable technologies such as Fitbits, wrist-worn epilepsy seizure detection devices, and other wearable accelerometers, together with smartphones, in remote measurement and assessment systems. Priority use cases for the further development and evaluation of RMT in epilepsy according to this study are: more accurate seizure records, automatically analyzing trends, improving triage through review of RMT data, and alerting patients and carers to imminent seizures. In depression, priority use cases are using RMT to detect trends or changes in the condition, monitoring treatment response and using data to inform treatment decisions, and self-management through monitoring and behavioral nudges. Some clinicians recognize the benefits of RMT in the management of MS by enabling the detection of the silent progression of the disease, detecting depression, and enabling the donation of data to biobanks, although clear priorities among these cannot be distinguished from our results.

The implementation of RMT will have different implications in different health service models. Cost-effectiveness studies will be required to understand the economic value of implementing RMT in clinical practice in different regions. Future work could also usefully explore the potential of RMT in other clinical conditions, as well as seeking to understand factors affecting adherence to remote measurement regimes in real-world conditions. Clinicians participating in this study considered passive data to be more reliable than active data, and further work is required to understand whether digital biomarkers based on passive remote measurement data can be used as proxies or replacements for existing measures in these and other clinical conditions. Overall, the Delphi method has been useful for prioritizing use cases and deriving insights into the practical application of RMT in clinical practice.