This review was conducted using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines (Multimedia Appendix 1) and was registered with the Prospective Register of Systematic Reviews (PROSPERO CRD42021228703). Our search was conducted using Ovid Embase and MEDLINE on May 11, 2022 (Multimedia Appendix 2 [16-22]). Based on the most globally prevalent and costly neurological disorders [11], studies investigating the use of NLP in Alzheimer disease (exclusive of Alzheimer disease–related disorders), Parkinson disease, stroke and transient ischemic attack, epilepsy, multiple sclerosis (MS), and migraine were included.

Studies that used NLP to analyze radiographic findings without any clinical correlation (eg, silent brain infarcts) or for purposes other than diagnosis, detection, phenotyping, subtyping, prognostication, risk stratification, or therapy were excluded. We excluded studies with populations comprised of patients with heterogeneous diseases or ambiguously defined populations (eg, we excluded studies that used a patient cohort consisting of patients with both Alzheimer dementia and mild cognitive impairment) as well as studies that did not use NLP for direct clinical applications. Examples of indirect clinical applications include the use of NLP to identify cohorts for subsequent model development or conduct epidemiological associations between cohorts without direct impact on clinical practice. We additionally excluded abstracts, conference proceedings, reviews, and editorials.

A medical librarian (SW) with expertise in scoping reviews first conducted a literature search (Multimedia Appendix 2) based on our eligibility criteria to generate a list of abstracts, which were then imported into a web application (Covidence Ltd) for initial screening by 3 authors (BRK, LJB, and IL). After the abstract screening was completed, full-text papers for screened abstracts were reviewed by 2 authors (BRK and IL) to determine eligibility for inclusion. Disagreements at both stages were resolved by discussion and consensus.

Using the final list of full-text studies, study characteristics were manually extracted by 1 author (IL) and charted in a REDCap (Research Electronic Data Capture; REDCap Consortium) web database form, which was subsequently reviewed by a second author (BRK) for accuracy. The data charting form was initially tested by the data extractor (IL) and revised after feedback from all coauthors (BRK, NJ, LJB, and SW). We extracted study publication year, population size, country of origin, journal field (eg, medical informatics, clinical neurology, nonclinical neuroscience, clinical medicine, or other), neurological disorder, and target of NLP (eg, diagnosis or detection, phenotyping or subtyping and severity, prognostication or risk stratification, or disease management or therapy). Each study could have multiple targets whenever applicable.

For each study, the source language to which NLP techniques were applied was also extracted. For studies conducted in or authored by teams from non-English–speaking countries, the source language was extrapolated directly as described from the study methodology. If the source language was a publicly available research dataset or ontology (eg, MetaMap ontology or ADReSS dataset, both of which use English), the source language was reported as English. Source of language for NLP (eg, clinical notes, radiographic reports, speech audio, or other) and type of study (eg, model derivation, validation, or both) were also noted. Validation studies were defined as studies that specifically investigated the validation of a derived model in a population external to the original model derivation population. Our definition of validation studies did not include validation on held-out test sets as part of model derivation. If the NLP model was both derived and externally validated in the same study, the population size included the additional population used for validation. Simulated patients, who were used as a training set in one study, were included in the population size. If no population size was mentioned in the studies, the number of text instances (eg, clinical notes and radiographic reports) was recorded.

We additionally extracted the study’s NLP approaches (ie, rule-based methods, ML, or both). Rule-based NLP included any approaches that used keyword searches, pattern matching, regular expressions, or ontological systems for word-concept mapping, text preprocessing, or classification. ML-based NLP comprised both conventional ML and DL approaches and both were distinguished as dichotomous study characteristic variables but could co-occur in the studies. A study was characterized as including any of these methods if either ML or DL was used at any point in model development for the study.

Under the category of conventional ML methods, linear regression, logistic regression, support vector machines (SVMs), naïve Bayes classifiers, decision trees, random forest classifiers, k-nearest neighbor algorithms, gradient boosting techniques such as extreme gradient boosting, latent Dirichlet allocation, and shallow neural networks were included. Under the definition of shallow neural network, we included any approaches using Word2vec or other “-2vec” word-embedding techniques that use a neural network to construct word contexts and extract semantic and syntactic meaning from text [23,24]. We also included other types of regression, such as lasso regression, which is often used for dimensionality reduction, in the conventional ML category.

DL techniques included convolutional neural networks, recurrent neural networks (RNNs), long- and short-term memory networks, multilayer perceptrons, and transformers. Studies using long- and short-term memory networks were also categorized as using an RNN. We also note that neural networks of unspecified type and number of layers, which were not clearly referred to as DL in the study, were not included in this category.

In total, 916 studies were identified from our search strategy, of which 271 were duplicates and were excluded. We then screened the resulting 645 abstracts, of which 565 were excluded due to not meeting initial eligibility criteria. Of the remaining 80 studies, 39 (49%) were excluded. The 2 most common reasons for exclusion were the use of NLP for nonclinical applications (n=15, 38%) and heterogeneous clinical populations (n=12, 31%). In total, 41 (4.5%) of the 916 studies from the original search results were ultimately included for review (Figure 1 and Table 1).

Of the 41 included studies, NLP was applied to stroke or transient ischemic attack in 20 (49%) studies, epilepsy in 10 (24%) studies, Alzheimer dementia in 6 (15%) studies, and MS in 5 (12%) studies. We found no studies applying NLP to Parkinson disease or migraine that met our eligibility criteria. Across all neurological conditions, NLP was most commonly applied for the purposes of detection or diagnosis (n=20, 49%), followed by clinical disease phenotyping or subtyping (n=17, 41%), prognostication or risk stratification (n=9, 22%), and management or therapy (n=4, 10%; Table 2).

Of the 41 studies, the language sources for NLP comprised clinical notes (n=25, 61%); radiology reports (n=14, 34%); speech (n=8, 20%); and other sources (n=2, 5%) that included echocardiography reports, letters to referring providers, and problem lists (Table 3). Of studies with speech as the language source, half (4/8, 50%) analyzed transcripts only, whereas half additionally incorporated acoustic features from the audio files themselves. These transcripts and audio files were largely from research datasets (eg, ADReSS and Pitt corpus). Two studies analyzed transcripts from interviews with patients. In the study including problem lists, it is unknown who reported the problems.

Of the 41 studies, the most common source language for NLP was English (n=39, 95%), Portuguese in 1 (2%) study, and unspecified in the remaining 1 study (which was of Chinese nationality, not multicentric). When patient population size was recorded, the median was 1091 (IQR 188-4211). In studies that did not specify a population size (n=4, 10%), the median number of clinical or radiographic notes was 2172 (IQR 1155.5-22,018.0).

Papers were most commonly published in medical informatics (n=15, 37%) journals, followed closely by clinical neurology (n=14, 34%) journals. Seven (17%) studies were published in nonclinical neuroscience journals; 2 (5%) in clinical medicine journals; and 1 (2%) each in neuroradiology, public health, and pharmacy journals. Studies were mostly conducted in the United States (n=21, 51%), followed by Taiwan (n=4, 10%) and the United Kingdom, Canada, and Australia (n=3, 7% each). Two (5%) studies were conducted in China, and 1 (2%) study was conducted in each of South Korea, Brazil, Iran, India, and Slovenia (Figure 2).

Only 6 (15%) studies used strictly rule-based methods. The majority of studies incorporated ML (n=35, 85%), either exclusively (n=18, 44%) or in combination with rule-based methods (n=17, 41%). Of the studies that used ML, most (n=31, 89%) used conventional ML methods, whereas 16 (46%) used DL approaches (Table 3), and 12 (34%) used a combination of both conventional ML and DL approaches.

As shown in Figure 3, the most frequently used conventional ML algorithms were random forest (n=18, 58%), SVM (n=15, 48%), and logistic regression (n=15, 48%) models. Among studies using DL approaches, transformers (n=10, 63%) were the most commonly used algorithm, followed by convolutional neural networks and RNNs (each n=7, 44%). The co-occurrence of random forest and transformer algorithms was a prevalent trend in research combining traditional ML with DL methodologies (n=6, 15%). Studies that used DL only began to appear in 2019 and later (Figure 4). The most often reported performance metrics for ML models were precision or recall (n=31, 76%), accuracy (n=22, 54%), area under the receiver operating curve (n=20, 49%), and F₁-score (n=19, 46%).

All 41 studies were model derivation studies, with only 7 (17%) studies conducting additional external validation (Multimedia Appendix 2). Furthermore, nearly all the study models were developed retrospectively and were not applied in practice or deployed in real-world environments, except for 3 studies. A study by Li et al [16] developed a model for stroke detection from imaging reports and then applied it to quantify the change in stroke cases before and during the COVID-19 pandemic. A second by Sung et al [53], also in the stroke category, evaluated the deployment of a user-interface system to determine intravenous thrombolysis eligibility built on the NLP model devised. A third study by Wissel et al [49] created a model to identify surgical resection candidates in adult patients with epilepsy. The model was retrained prospectively to incorporate new information.

In studies focused on Alzheimer dementia, diagnosis and detection was the only target of NLP (6/6, 100%). Disease phenotyping and subtyping was the most common purpose of NLP in stroke (10/20, 50%) and MS (4/5, 80%), whereas prognostication was seen as often as diagnosis in epilepsy studies (4/10, 40%; Figure S9 in Multimedia Appendix 2). Studies that applied NLP for the purpose of disease treatment or management were limited to stroke and epilepsy (Figure S9 in Multimedia Appendix 2).

Rule-based methods were used across all studies, except for Alzheimer dementia, in which only ML approaches were used (Figure S10 in Multimedia Appendix 2). Conventional ML methods were used most often by Alzheimer dementia studies (5/6, 83%), followed by stroke (16/20, 80%). Similarly, DL methods were used predominantly by Alzheimer dementia (6/6, 100%) and stroke (8/20, 40%) studies (Figure S10 in Multimedia Appendix 2). The transformer was the DL method used most frequently in Alzheimer disease-related studies (5/6, 83%).

In this scoping review, 41 studies [13,16-22,25-57] that investigated direct clinical applications of NLP to common neurological disorders were identified. We found that the majority of these studies focused on detection and diagnosis and applied NLP to stroke, whereas we found no studies of NLP that met our eligibility criteria in the clinical areas of migraine or Parkinson disease. Methodologically, ML techniques were used more often than rule-based methods, but a considerable number of studies still relied on rule-based approaches in combination with ML. While we observed that DL began to emerge as a methodology for NLP in 2019, we found that the transformer was the most commonly used DL algorithm overall.

At the time of writing, we believe our scoping review to be the first to examine direct clinical NLP applications in common neurological conditions. One prior review [58] investigated NLP applications across the combined clinical specialties of neurosurgery, spine surgery, and neurology, whereas another evaluated the use of NLP in both psychiatry and clinical neuroscience [59]. However, neither reviews analyzed studies and NLP applications according to neurological condition. More importantly, these reviews included many studies where NLP was not applied for direct clinical use, instead aiming to perform tasks such as characterizing patient cohorts [58], analyzing information extraction, or determining causal inference between concepts [59]. In contrast to this prior work, our review focused on direct clinical applications of NLP.

Of note, we found no studies applying NLP to migraine or Parkinson disease that met our eligibility criteria, thereby highlighting a potential gap in NLP research focusing on these disorders. This is perhaps unexpected, as the combined prevalence of migraine and Parkinson disease in the United States exceeds that of both stroke and MS [12]. Two explanations may account for this finding. One is that migraine and Parkinson disease may rely less on radiographic imaging studies and their reports to establish a diagnosis than stroke, Alzheimer dementia, or MS. Given that many ML applications in stroke have focused on neuroimaging [60], it is plausible that stroke imaging reports could represent an important source of data for NLP analyses. Indeed, the results of our review demonstrate that stroke-related NLP studies made use of radiographic reports as often as clinical notes for source text, which could have resulted in a relatively higher number of NLP studies within stroke than in other neurological conditions.

A second explanation may be that Alzheimer disease is a more common cause of dementia worldwide than dementing syndromes associated with Parkinson disease [61] and has in turn garnered a larger proportion of research funding. National Institutes of Health [62] research funding for Alzheimer dementia was approximately US $3 billion in 2022, as compared to US $259 million for Parkinson disease.

Our finding that NLP was most frequently applied to diagnostic problems is expected, given that clinical decision support is a common focus of artificial intelligence in medicine [63]. Historically, clinical decision support has also played an important role in medical informatics by constituting the main focus of archetypal systems such as MYCIN, INTERNIST-1, and DXplain, which were first developed in the 1970s and 1980s [64]. An alternative explanation is that the shortage of neurologists that already exists worldwide [65] may have potentially created a more urgent need for detection-oriented NLP applications rather than NLP applications targeting therapeutic management or prognostication.

Though diagnosis was the most common target of NLP overall, we found that epilepsy-related studies focused as much on prognostication as they did on diagnostic tasks. Given that roughly one-third of all patients with epilepsy are drug resistant [66], determining good surgical resection candidates as well as predicting surgical outcomes are important objectives that have been the focus of considerable research [67]. Consistent with this, the epilepsy-related studies in the prognostication category were directed toward identifying adult [49] and pediatric [42] surgical candidates, predicting postsurgical outcomes [43], and detecting risk factors for sudden unexpected death in epilepsy [17].

With respect to the types of ML models we found in our review, the relatively high proportion of conventional ML-based studies using random forest and SVM (18/31, 58% and 15/31, 48%, respectively) may have been related to the fact that SVM together with random forest models generally represented the dominant ML techniques prior to the advent of neural networks [68] in diagnostic and clinical decision support applications [63,69,70]. Despite its position as a potentially more basic classification method than either SVM or random forest, logistic regression was used as commonly as SVM in our analysis.

Furthermore, while we found that SVM and random forest models were common in ML-based NLP approaches, the optimal problems these models address are fundamentally different. SVM generally works best as a binary classifier, whereas random forest models are best used for classification tasks involving multiple categories [71]. We found that the most frequently used ML algorithms in stroke-related NLP studies were random forest models. This matches the most frequent target of NLP in stroke-related studies, which was disease subtyping (a multiple classification problem).

Among DL algorithms, which are becoming increasingly widespread in NLP [72], the transformer was the most commonly used technique we identified. Unlike other word embedding methods, a transformer processes a whole sequence of text while preserving the context and meaning of words [59,73]. Another significant advantage of transformers is that they can use transfer learning, which first trains a model on a learning task and then applies the model to a separate but closely related task [58,74]. A prevalent example of transfer learning in our results is Bidirectional Encoder Representations From Transformers (BERT), a transformer model that was originally trained using publicly available text from Wikipedia and BookCorpus, a collection of free, unpublished novels consisting of over 50 million sentences [75,76]. BERT can then be further refined on a target training task and dataset before being passed to a separate classification algorithm [28]. This is helpful in situations where the target training set is small [28]. The high frequency of Alzheimer disease–related NLP studies we found using BERT is expected within this context, as these studies often used the ADReSS speech dataset that consists of only 78 healthy controls and 78 patients with Alzheimer disease [28,45].

A particularly important finding of our review is that although many of the NLP studies leveraged powerful and sophisticated computational tools, most studies constitute research work rather than reports of operationalization or evaluation in practical settings. This is consistent with the current state of clinical NLP outside of neurology, wherein real-world deployment of NLP models continues to be limited [7,77,78].

One major obstacle to the implementation of NLP in clinical practice is model generalizability [7]. Published NLP models are usually internally validated rather than externally validated [7,17], limiting the understanding of model accuracy beyond the model’s original training environment [60]. We found this to be true for the majority of studies identified in our review. The lack of EMR standardization, including note formatting [17,78], documentation styles, and radiographic report structures across different medical institutions [7] and between clinicians, may partly account for our observations. Furthermore, the preponderance of English language as source text in NLP [79], as demonstrated by the single study in our review using non-English (Portuguese) text for analysis, suggests that the generalizability of NLP within neurology is most likely limited outside the English language.

Another major obstacle impeding the adoption of NLP tools is the inherent lack of transparency of ML-based algorithms [60], particularly artificial neural networks and other forms of DL approaches [80]. These approaches have low transparency because the computational methods they use to characterize relationships between inputs and outputs are not readily intelligible to humans [7,78,80] acting as a black box that could undermine clinicians’ trust in their performance.

The lack of well-defined regulatory guidelines and standards overseeing the artificial intelligence space [81] has furthered this mistrust. Compromise of personal health data, algorithmic bias, and the question of how to attribute culpability when diagnostic errors arise [82,83] are all ethical concerns that may serve to explain the relative paucity of studies across all neurological conditions that externally validated DL models.

Finally, the lack of portability of NLP applications into external EMRs is another factor that has restricted the development of NLP models to the research arena. External software modules containing ML and DL models are challenging to integrate into EMRs [1,84], as most implementations require a high level of computing infrastructure and technical expertise that many hospital information technology systems and personnel may lack [84]. Recent work suggests few EMR-integrated aggregative tools exist to display NLP findings to clinicians in a digestible format [85]. To address these barriers, some authors have advocated for collaborations between NLP researchers and EMR companies [77].

Our scoping review has several limitations. First, we note that the target of NLP was categorized according to author experience and interpretation of the literature, which may have underreported the application of the published NLP algorithms. Second, due to the variable performance metrics and outcomes across studies, we did not aggregate measurements of performance in our review, and we therefore could not reliably provide summary performance metrics for NLP models within individual diseases, applications, or outcomes. Future work should focus on individual outcomes within a clinical disorder for a more exact appraisal of NLP model performance than this review.

Third, this review only included studies based on common neurological disorders, direct clinical applications of NLP, and homogeneous clinical populations, which limited the number of studies we identified. It is therefore important to note that this review cannot be used to make definitive conclusions on the state of NLP research across all neurological disorders. Future efforts can be directed at characterizing the use of NLP across less common neurological disorders as well as in heterogeneous or ambiguously defined clinical populations. As NLP technologies continue to advance, it will also be critically important to evaluate studies that use newer transformers, such as GPT3, which have better performance than BERT models [59].

The abundance of unstructured text data in modern-day EMRs as well as the emphasis in neurology on narrative history and physical examination and heavy reliance on ancillary information such as radiographic reports and speech, all create an optimal use case for applying NLP for the diagnosis, management, or prognostication of neurological disorders. To our knowledge, this is the first attempt to systematically characterize research efforts to investigate direct NLP applications to common neurological conditions. Our review reveals gaps in neurological NLP research, showing a relative deficiency of NLP studies in subspecialties outside of stroke or epilepsy, and underlines the need to actualize NLP models outside of the research phase. Moreover, the current emphasis of NLP on diagnostic tasks suggests that NLP may be particularly useful in settings that lack access to neurological expertise.